Shell Programming and Scripting

Hi Everyone,

I am trying to write a shell script that can find the longest phrase that appears at least twice in an online news article. The HTML has been parsed through an HTML parser, converted to XML and the article content extracted. I have put this article content in a text file to work from. Eventually, this will be read from the database.

Right now, I need to find the longest phrase that appears in this text file at least twice. I can extract bigrams and trigrams but does anyone know of a way to sift through the file and find the longest phrase, irrespective of it being an n-gram ?

Thanks. Much appreciated
SG

Maybe you can post a sample.

The text file is nothing but a news article after parsing HTML tags and extracting the content using XML. I use the following to extract and print the most bigrams for now.

tr -sc 'a-zA-z0-9.' '\012' < $1 > bigrams1
tail -n+2 bigrams1 > bigrams2
paste bigrams1 bigrams2

Here $1 is the name of the file containing the actual text. I am passing this as an argument for now.

Thing is, detecting bigrams and trigrams is easy. Is there a way to detect the longest phrase that appears at least twice ? It could be a bigram, a trigram or n-gram.

Thanks.
SG

As an idea .. I was thinking of maybe counting the total number of words in the text file and then running a for loop to that number to check for n-grams.

I haven't yet tried this idea. Right now I am working out of text files. Creating 2 text files for bigrams is easy .. but in real time if I have to create n files when checking for n-grams ... how feasible is that in terms of memory and CPU cycles both ??

This is why I am asking here, if any of you can help me please

Thanks
SG

I think you need to define what a 'phrase' is.

A phrase is a collection of n number of words that are the same. Could be anything from 2 to n. So for example in a sample text:

The quick brown fox jumped over the ugly dog
The quick brown fox is fast asleep in a corner.

In the above sample 'The quick brown fox' is a the phrase I am looking for, as it occurs twice in the text. In another sample

My name is John Doe
My name used to Alfred

The "phrase" here will be "My name" as that is what is consistent.

I hope that makes a bit more sense ?

Thanks
SG

Makes sense, not that I want to figure out how to do it.