Hi all,

I need some help to remove duplicates from a file before merging.

I have got 2 files:

file1 has data in format
4300 23456
4301 2357

the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file

file 2 has data in same format but is not in sorted order

3999 2349
4305 1234
4300 3459

the script i am using is not able to remove duplicate values after spaces, and the output is coming in the following manner:-

3999 2349
4300 233445569
4301 2357
4305 1234

The expected output is:-
3999 2349
4300 234569
4301 2357
4305 1234

The script i am using is:-

Code:

export INPUT1=File1
export REQ=File2
export OUTPUT1=Final_File


sort $INPUT1 $REQ |
           awk     '$1 == SV1              {SV2=SV2  $2;next}
           NR>1 && $1 != SV1                {print SV1, SV2}
                                            {SV1=$1;SV2=$2}
           END                             {print  SV1,SV2}
        ' > $OUTPUT1