Shell Programming and Scripting

Hi,

I have a .txt file (uniqfields.txt) with 3 fields separated by " | " (pipe symbol). This file contains unique values with respect to all these 3 fields taken together. There are about 40,000 SORTED records (rows) in this file. Sample records are given below.

Code:

1TVAO|OVEPT|VO
1TVAO|OVPDM|VO
6NFXE|17CLP|DH
6NFXE|NRZO4|EQ
6NFXE|SMOSA|EQ
ACA15|11X1W|DX
ACA15|1LN88|DX
ACA15|1LNSK|DX
ACA15|1LNVX|DX
ACA15|1LNVX|FD

Now, there is another file (mainfile.txt) which contains 23 fields(columns), which contains the above fields as 7th, 13th and 14th field respectively. This file ie, mainfile.txt is also sorted and is seperated by pipe symbol. This file contains about 77,000 records. The 7th, 13th & 14th columns are from with in the above values only, but some records(rows) are repeated (with respect to these 3 fields(columns), other fields(columns) may or maynot be same.

What i need to do now is to compare first record (1st row) of uniqfields.txt with that of mainfile.txt and fetch first record (row) which contains all the above 3 fields same. That is, 1st field from uniqfields.txt should match with 7th column of mainfile.txt AND 2nd of uniqfields.txt with 13th of mainfile.txt AND 3rd of uniqfields.txt with 14th of mainfile.txt.

Why this is required with awk script..??

1) As i'm new to unix, i'm just catchin up wit awk and i'm not able to find a solution myself.
2) I tried with sort -t\| -u +6 -7 +12 -14 mainfile.txt > uniqmainfile.txt, but its working fine in SunOS and not working on NCR MP-RAS.
Actually, we are migrating these from server with SunOS to server with NCR MP-RAS. So, in SunOS, if i run the sort script just mentioned above, its fetching the first unique record, whereas in NCR MP-RAS its fetching the last unique record.

How does it impact if it takes last or first among from similar records..??

Actually, as we are checking for uniqueness only in the 7th, 13th and 14th fields(columns), the other fields(columns) are not matched with the reports in MP-RAS with that of SunOS.

I've tried with -r and tried using uniq command as well,but in vain. And found that the only solution is using awk.

Please help me in this regard.

Thanks,
RRVARMA

How about this:

Code:

awk -F '|' '
        # load uniqfile.txt into array
        NR==FNR { uniq[$1,$2,$3]=1; next }
        # print record if index is present in uniq array, change value to prevent 
        # printing duplicate records
        uniq[$7,$13,$14] == 1 { print; uniq[$7,$13,$14]=0 }
' uniqfile.txt mainfile.txt

I'm not sure how well it will perform since it loads the entire uniqfile.txt into an array, so it might use a significant amount of memory.

Hi Annihilannic,

Its showing this error..

Code:

awk: can't open |

As i said.. i'm not that good in awk.. Could you please suggest some other way.

Thanks Annihilannic,
RRVARMA