Hi, I've been trying to removed duplicates lines with similar columns in a fixed width file and it's not working.
I've search the forum but nothing comes close.

I have a sample file:

27147140631203RA CCD *
27147140631203RA PPN *
37147140631207RD AAA
47147140631203RD JNA
47147140631204DC ADK *
47147140631204DC ALK *
67147140631203DA ALM *
67147140631203DA CCD *
77147140631209QC RRP
87147140631203QA RRN

There are 3 spaces between first set of alphanumerics and the last three letter codes.

I want to remove lines that match only up to the 3 blanks and ignore the 3 letter codes or whatever else is on that line after the 3 letter codes.

Anyone know how I can do this? I want to keep at least one instance of any duplicates...doesn't matter which.
I put asteriks where I need to keep one of any two.

Thanks.
Gianni

assuming the first field is always 16 chars you can:

uniq -w16

I tried different combinations of sort and uniq, etc but none worked.
Also, I am on AIX and korn shell. When I ran uniq -?, I got:

uniq: Not a recognized flag: ?
Usage: uniq [-c | -d | -u] [-f Fields] [-s Chars] [-Fields] [+Chars] [InFile [OutFile]]

I have no -w switch...

Thanks.

right so your uniq can only skip fields or chars.
How about swaping the fields using sed like:

sed 's/\([^ ]*\) *\(.*\)$/\2 \1/' |
uniq -f1
sed 's/\([^ ]*\) *\(.*\)$/\2 \1/'

Try:
sort -mu -k1,1 < datafile

Or an even more cryptic version:
All this does is create an associative array. The first time it encounters the array element it will be zero, so it will print the whole record. If the element is not zero we have seen it before, so do not print it. $1 is the first field in the record.