Shell Programming and Scripting

Dear all,
I have a large database of names which I have sorted on reverse with a Perl Script. A sample is provided below

Code:

agarsingh
aghansingh
akalsingh
akamsingh
akbareesingh
akhamisingh
akramysingh
akuvsingh
anchalusingh
andaroosingh
angadsingh
anjawsingh
angibai
angobai
angurbai
angureebai
anjabai
anjiqbai
anjsbai
anjanybai
anjatibai
anjopbai
anjusbai
ankikbai
akhileshkumar
akleskumar
akshaykumar
anchalkumar
anjanikumar
ankitkumar
antimkumar

My problem is that I wish to identify the suffixes ( i.e. the possible identical longest strings moving from right to left) which are adjoined to the names and store such strings along with their frequency in a separate file with the following conditions
the suffix string should be at least between 3 and 5 characters in length
the suffix string should be repeated at least 10 times in the database.

Thus in the sample given above, the script would identify only the following suffixes along with their frequency

Code:

singh	12
bai	12

The suffix

Code:

kumar

Will not be identified since it is less than 10 times.
I had posted the query earlier, but at present I have tried to refine it with conditional constraints so that hopefully only the most pertinent suffixes will be identified. There could be a few false positives but I could weed them out.
I work in a Windows environment and PERL or AWK script would be helpful.
Many thanks and all good wishes for the New Year to all the folks who take their valuable time off to help people solve their problems

For starters, this:

Code:

$ awk '{while($1=substr($1,2)) if (length($1)>=3) A[$1]++} END{for(i in A) if(A[i]>=10) print i, A[i]}' file 
ngh 12
ingh 12
bai 12
singh 12

would produce the names with some false positive substrings .

Under Windows you probably need to put the script in a script file

Code:

{
  while($1=substr($1,2)) if (length($1)>=3) A[$1]++
} 
END {
  for(i in A) if(A[i]>=10) print i, A[i]
}

and run it as

Code:

awk -f script_file inputfile

Many thanks. It worked very well. When I posted the request, I knew that there are chances of false positives, but a list of suffixes is easier to handle than wading through thousands of lines.
I can also tweak the awk script if I wish to set the range
Happy New Year and thanks once more

While the earlier method worked and I had to tweak a few suffixes manually, I have been rethinking the process of identification of suffixed names and after going through nearly 40 to 50 thousand names, I have identified a pattern. Very often, in nearly 95% of the cases,the name that is suffixed is also a name by itself as in the example below and comes first in my rev sort followed by names to which it is suffixed.

Code:

singh
agarsingh
aghansingh
akalsingh
akamsingh
akbareesingh
akhamisingh
akramysingh
akuvsingh
anchalusingh
andaroosingh
angadsingh
anjawsingh
bai
angibai
angobai
angurbai
angureebai
anjabai
anjiqbai
anjsbai
anjanybai
anjatibai
anjopbai
anjusbai
ankikbai
kumar
akhileshkumar
akleskumar
akshaykumar
anchalkumar
anjanikumar
ankitkumar
antimkumar

Could it be possible to extract such suffixes given that the suffix is a stand-alone name as in the case of

Code:

kumar
bai
singh

with the proviso that the standalone name is suffixed at least three times to another name. This would obviate the need for blind search and also false positives. I know that this could possibly miss out a few suffixes, but from my analysis, this could provide a more accurate solution.
Would it be possible to devise a PERL or AWK script to identify such cases.
Many thanks once again for all kind help.

How about

Code:

awk '
        {if ($0 !~ IX "$" || NR == 1) IX = $0
         else CNT[IX]++
        }
END     {for (c in CNT) print c, CNT[c]
        }
' file
kumar 7
bai 12
singh 12

Thanks a lot. It works well, all I had to do was trim off short words from the list and which in no way were suffixes, and I managed to get a pretty comprehensive lst of suffixes.
I have been studying the syntax of the script and there is one part which perplexes me. The rest I could grab

Code:

NR == 1

Could you please explain what this really does.
Thanks once again and a Happy New Year.

NR is the record counter, so this condition is true on the first line of the input stream.