Removing duplicates - The UNIX and Linux Forums

		The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
Removing duplicates

google unix.com

Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
finding duplicates in columns and removing lines	totus	Shell Programming and Scripting	17	11-29-2008 11:27 AM
removing duplicates based on key	pukars4u	Shell Programming and Scripting	1	05-21-2008 03:50 PM
removing duplicates from a file	trichyselva	UNIX for Dummies Questions & Answers	2	03-25-2008 10:49 AM
removing duplicates and sort -k	orahi001	UNIX for Dummies Questions & Answers	3	01-25-2008 09:59 AM
Removing duplicates [sort , uniq]	sharatz83	Shell Programming and Scripting	4	07-14-2006 05:12 PM


	Powered by

LinkBack

Thread Tools

Search this Thread

Rate Thread

Display Modes

#7 (permalink)

09-14-2005

jim mcnamara jim mcnamara is offline

...@...

Join Date: Feb 2004

Location: NM

Posts: 5,759

Or an even more cryptic version:

Code:

awk '!x[$1]++' filename > newfile

All this does is create an associative array. The first time it encounters the array element it will be zero, so it will print the whole record. If the element is not zero we have seen it before, so do not print it. $1 is the first field in the record.

Bookmarks

« Subtract 100 from first field in long list? Simple manipulation? | sed help again »

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Switch to Hybrid Mode Threaded Mode	Rate This Thread:

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Trackbacks are On Pingbacks are On Refbacks are On Forum Rules

All times are GMT -4. The time now is 02:24 PM.