Extract all the sentences from a text file that matches a pattern list
Hi
I have a big text file. I want to extract all the sentences that matches at least 70% (seventy percent) of the words from each sentence based on a word list called A.
Say the format of the text file is as given below:
The output format is based on the availability of all the words from the words list A. If the sentence matches all the words from the word list, then the sentence will be extracted. The condition is that at least seventy percent of the words in the sentence must be matched from the words list A.
Assuming that all the capital letter words such as AAA, BBB, CCC, KKK, XXX, YYY, ZZZ, DDD, FFF, EEE, GGG and HHH are not found in the words list A. After the extraction, the output will look like as given below.
I need help to write a script for the above problem. A sample script will be really helpful to me. Thanks in advance.
Last edited by Don Cragun; 06-25-2015 at 03:08 AM..
Reason: Change HTML tags to CODE tags.
This is not an assignment at all. This is for my personal interest for processing text with different coding.
I have tried this using C code instead of scripts but I would prefer script because they are comparatively faster and convenient to put in the pipeline. Moreover, unix scripts are convenient for processing text. Bash shell is the one I use.
There is no actual sample of word list or token list.
To clarify, I would say tokens instead of words and sentence as a set of tokens. I use Ubuntu OS.
Please let me know if you need more details.
Thanks in advance.
Let me say first that there's incredibly refined and sophisticated algorithms out there, used by e.g. the various search engines to analyse all the internet sites around the globe and to hand you the results in a split second, so anything posted here is a clumsy approach cobbled together without any optimisation. Anyhow, try
You may want/need to get rid of punctuation first in a real world sample.
Changing "word" to "token" and "sentence" to "set" doesn't clarify anything. Changing two undefined terms to two other undefined terms still leaves us with no defined terms. If you refuse to explain what you want your code to do, there is no reason for any of us to waste our time trying to guess at your requirements, nor to try to write code when we don't know what the code is supposed to do. Why did you explicitly say that you had a "word list A" if there is no word list? Your original requirement was:
Quote:
The output format is based on the availability of all the words from the words list A. If the sentence matches all the words from the word list, then the sentence will be extracted. The condition is that at least seventy percent of the words in the sentence must be matched from the words list A.
which can now be restated as:
Quote:
The output format is based on the availability of all of the <undefined term1>s from the <undefined term1>s in a non-existent list. If the <undefined term2> matches all of the <undefined term1>s from the non-existent list, then the <undefined term2> will be extracted. The condition is that at least 70% of the <undefined term1>s in the <undefined term2> must be matched from the <undefined term1>s in the non-existent list.
With requirements like this, it looks like a homework assignment that you want us to complete for you.
If you already have a way to do this and just want to write it in a different language, show us the C code that you have written that you now want to translate to shell code. They we would be able to deduce your definitions from you C code and know what it is that we're trying to do. (But, don't claim that you are converting from C to shell to make the code faster; for any particular task, well crafted C will almost certainly be faster than a corresponding shell script. And, there is absolutely no reason to claim that C code can't be used in a pipeline. Almost all of the standard utilities on UNIX and Linux systems are written in C and many of them are perfectly capable of being used in a pipeline. Changing C code that can't be used as a filter to a shell script won't magically turn it into a filter.)
RudiC made a valiant effort to help you get a start on your problem, but it ignores the fact that you don't have a list, assumes that tokens (or words) include punctuation, assumes that <sentence> or <set> and <line in a text file> are synonymous, ignores the requirement to ignore uppercase <words> or <tokens> from you nonexistent list, and uses 80% instead of 70% as the threshold.
Last edited by Don Cragun; 06-25-2015 at 04:42 AM..
Reason: Fix typo and note difference in percentages.
This User Gave Thanks to Don Cragun For This Post:
hi all,
trying this using shell/bash with sed/awk/grep
I have two files, one containing one column, the other containing multiple columns (comma delimited).
file1.txt
abc12345
def12345
ghi54321
...
file2.txt
abc1,text1,texta
abc,text2,textb
def123,text3,textc
gh,text4,textd... (6 Replies)
Hi,
I have to extract the whole set if a pattern matches.i have a file called input.txt
input.txt
------------
CREATE TABLE ABC
(
A,
B,
C
);
CREATE TABLE XYZ
(
X,
Y,
Z,
P,
Q
); (6 Replies)
Hi
I have two text files. The first file is TEXTFILEONE.txt as given below:
<Text Text_ID="10155645315851111_10155645333076543" From="460350337461111" Created="2011-03-16T17:05:37+0000" use_count="123">This is the first text</Text>
<Text Text_ID="10155645315851111_10155645317023456"... (7 Replies)
Hi
This is my first post and I'm just a beginner. So please be nice to me.
I have a couple of html files where a pattern beginning with "http://www.site.com" and ending with "/resource.dat" is present on every 241st line. How do I extract this to a new text file?
I have tried sed -n 241,241p... (13 Replies)
I sat down yesterday to write this script and have just realised that my methodology is broken........
In essense I have.....
----------------------------------------------------------------- (This line really is in the file)
Service ID: 12345 ... (7 Replies)
I have an xml file with IP addresses all over the show. I want to print only the IP addresses and cut off any text before or after the IP address.
Example:
Note: The IP addresses (x.x.x.x) do not consistently appear in the xml file as per the pattern below. Sometimes there are text before... (8 Replies)
Hi All,
I am trying to extract data from a large text file , I want to extract lines which contains a five digit number followed by a hyphen , like
12345- , i tried with egrep ,eg : egrep "+" text.txt
but which returns all the lines which contains any number of digits followed by hyhen ,... (19 Replies)
This is my first post, please be nice. I have tried to google and read different tutorials.
The task at hand is:
Input file input.txt (example)
abc123defhij-E-1234jslo
456ujs-W-abXjklp
From this file the task is to grep the -E- and -W- strings that are unique and write a new file... (5 Replies)
Hi ,
i have a text file that contain a story
How do i extract the out all the sentences that contain the word Mon. in C++
I only want to show those sentences that contain the word mon
eg.
Monkey on a tree.
Rabbit jumping around the tree.
I am very rich, I have lots of money.
Today... (1 Reply)
Hi All,
I have an input below. I tried to use the awk below but it seems that it ;s not working. Can anybody help ?
My concept here is to find the 2nd field of the last occurrence of such pattern " ** XXX ccc ccc cc cc ccc 2007 " . In this case, the 2nd field is " XXX ". With this "XXX" term... (20 Replies)