Dear all,
I need a little help. I am working on a frequency driven database in which the structure is as under:

Code:

headword=gloss<space>Frequency

The data which I am working with has dupes i.e. the Headword is repeated more than once with a different gloss variant on the right hand side and also with a different frequency, as in the pseudo example given below:

Code:

John=Jean 1200
John=Jehan 300
John=Jan 1100
John=Johann 22

I have written a perl script which extracts all such instances. Assuming [I know that some may not like this approach] that the headword with the highest frequency is possibly the right candidate, I want to merge the ones with the lower frequency into the higher frequency with the caveat that if the difference in frequency is lower than 10% both be retained as such. Thus in the example provided the output would be as under:

Code:

John=Jean 1200
John=Jean 300
John=Jan 1100
John=Jean 22

Code:

John=Jean and John=Jan

are not merged since the deviation range is less than 10%. Others are merged since the deviation range is more than 10%.

I have been able to identify through a Perl Script all such instances of Dupes and also have a Script which merges frequencies when both headword and gloss are identical but in spite of all attempts cannot get the frequency problem to solve itself, especially calculating the deviation range. Since I am still a newbie, I cannot just get the operation right
Basically the steps are as under:

Code:

1. Identifying dupes [Already done]
2. Collecting all dupes i.e. the headword is identical in one list
3. Checking frequency
4. Identifying highest frequency in the list
5. Calculating frequency deviation range in the list. 
6. If more than 10% merging the lower frequency to the higher frequency i.e replacing the lower frequency gloss by the higher frequency gloss
7. If the deviation range is less than 10%; do not modify.
8. Finally merge frequencies of all words generated out by step 6 i.e. headword and gloss are identical [Already done]

In case someone would like to tangle with live data, here is a small sample:

Code:

trimbak=ત્રિંબક 87
trimbak= ત્રીંબક 35
trimbak=ત્રીંબક 14
trimbakbhai= ત્રીંબકભાઈ 55
trimbakbhai=ત્રિંબકભાઈ 7
trimbakbhai=ત્રીબંકભાઇ 4
tripathi=ત્રિપાટી 7
tripathi=ત્રિપાઠી 369
tripathi=ત્રિપાથી 4
tripathi=ત્રીપાઠી 8
trivedi=ત્રિવેદિ 28
trivedi=ત્રિવેદી 78
trivedi=ત્રીવેદી 8
trupti=તૃપ્તિ 4
trupti=તૃપ્તી 13

I have checked on the forum but the scripts there do not answer what I need. Many thanks for all help.