Dear All,

I have a file which I want to get the list of frequency of each word, ignoring list of stop words and now I have problems which punctuations and " 's ".

what I am doing is:

sed 's/[^a-zA-Z ]//g' file01.txt > file01-clear.txt
cat file01-clear.txt | tr "[:upper:]" "[:lower:]"| tr ' ' '\012' |sort |uniq -c |sort -n -r -k 1 > file01-FQ.txt
grep -v -F -f rejectfile.txt file01-FQ.txt > file01-results.txt

useless use of cat

awk 'NR==FNR { STOP[$1]++; next }

{
        gsub(/[^a-zA-Z \t]/, ""); # Replace non-whitespace with nothing.
        $0=tolower($0); # Lowercase and separate the string
        for(N=1; N<=NF; N++) if(!($N in STOP)) W[$1]++ # Count all non-stopwords
}

END { for(X in W) print W[X], X; } # Print count of words' stopwords.txt words.txt

First off, it might help to post a sample of your text (especially the one file with 26 words which is boiled down to one) because for most of us it is easier to understand what the problem is when we see it.

Second: you have hit the first real problem of all regexp constructors, which is: "what is a word"? Lets start.

The first attempt is what i found as a definition in IBM's "Handbook of Data Processing" some 30 years ago, which said:

A word is a sequence of nonblank characters separated by blanks.

To put it this in regexp (i write "<spc>" and "<tab> to denote space and tab characters, whitespace is hard to read in postings):

/[<spc><tab>]\([a-zA-Z][a-zA-Z]*\)[<spc><tab>]/

/[<spc><tab>]*\([a-zA-Z][a-zA-Z]*\)[<spc><tab>]*/

/[<spc><tab>]*\([a-zA-Z][a-zA-Z]*\)[<spc><tab>,;:.!]*/

/[<spc><tab>"']*\([a-zA-Z][a-zA-Z]*\)[<spc><tab>,;:.!"']*/

s/-/<spc>/g
/[<spc><tab>"']*\([a-zA-Z][a-zA-Z]*\)[<spc><tab>,;:.!"']*/

thanx for your help

problem solved