Finding the most frequently occurring set of words
Hi guys, I have a file with a list of phoneme for words, it looks like this:
Code:
AILS EY1 L Z
AIMLESSLY EY1 M L AH0 S L IY0
AIMONE EY1 M OW2 N
AIMS EY1 M Z
AINGE EY1 NG
AINGE(2) EY1 N JH
AINLEY EY1 N L IY0
AINSLIE EY1 N Z L IY0
AIR EH1 R
AIRBAGS EH1 R B AE2 G Z
and I need to find the phoneme pattern that corresponds to the largest number of words.
So for the above section of the list, the output would be
Code:
EY1 N
because it occurred three times. How should I do this?
Also, how can I find the line with the most phonemes in this list? The output just needs to be the number of phonemes and the actual line, like:
$ echo 'AIMLESSLY EY1 M L AH0 S L IY0' | awk '{for(i=2;i<NF;i++) {s=$i; for(j=i+1;j<=NF;j++) print s=s FS $j}}'
EY1 M
EY1 M L
EY1 M L AH0
EY1 M L AH0 S
EY1 M L AH0 S L
EY1 M L AH0 S L IY0
M L
M L AH0
M L AH0 S
M L AH0 S L
M L AH0 S L IY0
L AH0
L AH0 S
L AH0 S L
L AH0 S L IY0
AH0 S
AH0 S L
AH0 S L IY0
S L
S L IY0
L IY0
A list of of the occuring patterns sorted from most most common to least, ties sorted in increasing alphabetical order can be done with:
Code:
$ awk '{for(i=2;i<NF;i++) {s=$i; for(j=i+1;j<=NF;j++) print s=s FS $j}}' data | sort | uniq -c | sort -k1,1rn -k2
3 EY1 M
3 EY1 N
3 L IY0
2 EH1 R
1 AE2 G
1 AE2 G Z
1 AH0 S
1 AH0 S L
1 AH0 S L IY0
1 B AE2
1 B AE2 G
1 B AE2 G Z
1 EH1 R B
1 EH1 R B AE2
1 EH1 R B AE2 G
1 EH1 R B AE2 G Z
1 EY1 L
1 EY1 L Z
1 EY1 M L
1 EY1 M L AH0
1 EY1 M L AH0 S
1 EY1 M L AH0 S L
1 EY1 M L AH0 S L IY0
1 EY1 M OW2
1 EY1 M OW2 N
1 EY1 M Z
1 EY1 N JH
1 EY1 N L
1 EY1 N L IY0
1 EY1 N Z
1 EY1 N Z L
1 EY1 N Z L IY0
1 EY1 NG
1 G Z
1 L AH0
1 L AH0 S
1 L AH0 S L
1 L AH0 S L IY0
1 L Z
1 M L
1 M L AH0
1 M L AH0 S
1 M L AH0 S L
1 M L AH0 S L IY0
1 M OW2
1 M OW2 N
1 M Z
1 N JH
1 N L
1 N L IY0
1 N Z
1 N Z L
1 N Z L IY0
1 OW2 N
1 R B
1 R B AE2
1 R B AE2 G
1 R B AE2 G Z
1 S L
1 S L IY0
1 Z L
1 Z L IY0
If only the very first result is of interest append a head filter to the previous pipeline:
Code:
$ awk '{for(i=2;i<NF;i++) {s=$i; for(j=i+1;j<=NF;j++) print s=s FS $j}}' data | sort | uniq -c | sort -k1,1rn -k2 | head -n1
3 EY1 M
If the count is of no interest:
Code:
$ awk '{for(i=2;i<NF;i++) {s=$i; for(j=i+1;j<=NF;j++) print s=s FS $j}}' data | sort | uniq -c | sort -k1,1rn -k2 | sed 's/ *[^ ]* *//;q'
EY1 M
Hi
I have two files. One is a text file consisting of sentences i.e. INPUT.txt and the second file is SEARCH.txt consisting of two or three columns. I need help to write a script to search the second column of SEARCH.txt for each set of five words (blue color as set one and green color as set... (6 Replies)
Hi ,
I need to count the number of errors associated with the two words occurring in the file. It's about counting the occurrences of the word "error" for where is the word "index.js". As such the command should look like. Please kindly help. I was trying: grep "error" log.txt | wc -l (1 Reply)
Hi All,
I am completely stuck here.
I have a set of files (with names A.txt, B.txt until L.txt) which contain words like these:
computer
random access memory
computer networking
mouse
terminal
windows
All the files from A.txt to L.txt have the same format i.e. complete words in... (2 Replies)
Got a question for you guys...I am searching through a public directory (that has tons of files) trying to find a file that I was working on a longggggg time ago. I can't remember what it is called, but I do remember the content. It should contains words like this:
Joe
Pulvo
botnet
zeus... (5 Replies)
Hi All,
I tried this but I am having trouble formulating this:
I have a file that looks like this (this is a sample file words can be different):
network
router
frame
network
router
computer
card
host
computer
card
One can see that in this file "network" and "router" occur... (3 Replies)
i want to write a shell script that correct a text file.for example if i have the input file:
"john has has 2 apples
anne has 3 oranges oranges"
i want that the output file be like this:
"john has 2 apples
anne has 3 oranges"
i've tried to read line by line from input text file into array... (11 Replies)
Greetings.
I am a UNIX newbies.
I am currently facing difficulties dealing with a large data set and I would like to ask for helps. I have a input file like this:
ak 1 AAM1
ak 2 AAM1
ak 3 AAM1
ak 11 AMM2
ak 12 AMM2
ak 13 AMM2
ak 14 AMM2
Is there any possibility for me to... (7 Replies)
I need Display the most top 10 occurring words along with the number of occurences of those words in the given text.
Sample text as below:
"The Travails of Single South Indian men of conservative upbringing" or "Why we don't get any..."
Yet another action packed weekend in Mumbai, full of... (2 Replies)
hi,
cud u help me to find this.
i hav 2 files.
file1 has data as "ARUN
ARUN is from Australia
Arun likes America etc..
ARUN ARUN "
file2 has "ARUN
Australia
America"
i... (5 Replies)
03-21-2010
