Uniq using only the first field

Digby · #1 (**permalink**) 01-15-2008

Hi all,

I have a file that contains a list of codes (shown below).
I want to 'uniq' the file using only the first field. Anyone know an easy way of doing it?

Cheers,

Dave

##### Input File #####
1xr1 1xws 1yxt 1yxu 1yxv 1yxx 2o3p 2o63 2o64 2o65
1xr1 1xws 1yxt 1yxv 1yxx 2o3p 2o63 2o64 2o65
1xr1 1yxt 1yxv 1yxx 2o3p 2o63 2o64 2o65
1y8y 2c68 2c69 2c6i 2c6k 2c6m 2c6o
1y8y 2c69 2c6i 2c6k
1zlt 2br1 2brb 2brn 2bro 2c3k 2cgu 2cgv 2cgx
1zlt 2br1 2brb 2brn 2bro 2cgu 2cgv 2cgx
1zlt 2br1 2brb 2brn 2bro 2cgv 2cgx
1zlt 2br1 2brb 2brn 2bro 2cgx
1zlt 2br1 2brb 2bro

##### Desired Output File #######
1xr1 1xws 1yxt 1yxu 1yxv 1yxx 2o3p 2o63 2o64 2o65
1y8y 2c68 2c69 2c6i 2c6k 2c6m 2c6o
1zlt 2br1 2brb 2brn 2bro 2c3k 2cgu 2cgv 2cgx

radoulov · #2 (**permalink**) 01-15-2008

Yes:

Code:

awk '!x[$1]++' filename

Use nawk or /usr/xpg4/bin/awk on Solaris.

Digby · #3 (**permalink**) 01-15-2008

Thanks for the quick response radoulov.
It works a treat!

Tytalus · #4 (**permalink**) 01-15-2008

The neat nawk solution will not always provide the same solution independent of line ordering though...

something like:

Code:

awk '{if ( length($0) > length(a[$1]) ) {a[$1]=$0}} END {for (i in a){print a[i]}}' file

will ensure you get the longest line (if that is a requirement).

smallu · #5 (**permalink**) 01-15-2008

Tytalus

Can you pls explain how it works..whats $1 and $0 and how does it work..Thanks a bunch!

Tytalus · #6 (**permalink**) 01-16-2008

Quote:

Originally Posted by smallu

Tytalus

Can you pls explain how it works..whats $1 and $0 and how does it work..Thanks a bunch!

Sure.

$1 is the first field. $0 is the entire record. So...

Basic jist is that you run through each record, and add it to an array (using the first field as a reference) IF the length of the record is greater than the current array entry:

Code:

if ( length($0) > length(a[$1]) ){a[$1]=$0}}

Then you just loop through and print out the contents of the array:

Code:

{for (i in a){print a[i]}}'

HTH

ilan · #7 (**permalink**) 01-16-2008

simply it would be like...

uniq -w 4 filename

note: the catch here is that assumes your first feild would be always 4 :-)

-ilan

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
using uniq and awk??	amatuer_lee_3	Shell Programming and Scripting	13	05-17-2008 02:10 PM
How to uniq third field in a file	babycakes	UNIX for Dummies Questions & Answers	1	02-01-2008 03:52 AM
help on UniQ	vishal_ranjan	HP-UX	0	06-21-2007 11:33 AM
Moving Part of a field to another field using AWK	rjsha1	Shell Programming and Scripting	5	08-04-2006 06:39 AM
sort/uniq	jimmyflip	UNIX for Dummies Questions & Answers	3	10-17-2002 06:09 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread: