Sort, Uniq, Duplicates

Amruta Pitkar · #1 (**permalink**) 05-16-2007

Input File is :
-------------
25060008,0040,03,
25136437,0030,03,
25069457,0040,02,
80303438,0014,03,1st
80321837,0009,03,1st
80321977,0009,03,1st
80341345,0007,03,1st
84176527,0047,03,1st
84176527,0047,03,
20000735,0018,03,1st
25060008,0040,03,

I am using the following in the script :
------------------------------------
cat InputFile | sort -t, -k1,2 | uniq -d > "Duplicates"

This gets 25060008,0040,03, into the Duplicates file.
But I also want 84176527,0047,03, in the Duplicates file.

Basically I want the script to sort on the first 2 fields (delimited by comma) and if duplicates are found for first 2 fields I want it to be written to "Duplicates" file.

Please guide.

aigles · #2 (**permalink**) 05-16-2007

Try that:

Code:

sort -t, -k1,2 InputFile | awk -F, '{ if ((key=$1 "," $2)==prv_key) print; prv_key=key}' > "Duplicates"

Jean-Pierre.

matrixmadhan · #3 (**permalink**) 05-16-2007

25060008,0040,03,

this is the only line that is duplicate

matrixmadhan · #4 (**permalink**) 05-16-2007

Quote:

This gets 25060008,0040,03, into the Duplicates file.
But I also want 84176527,0047,03, in the Duplicates file.

Basically I want the script to sort on the first 2 fields (delimited by comma) and if duplicates are found for first 2 fields I want it to be written to "Duplicates" file.

In the above sample of records only the third field is common '03'
and not the first or the second field.

How would you expect that to be termed as duplicates based on two fields ?

Amruta Pitkar · #5 (**permalink**) 05-16-2007

Hi MatrixMadhan,
Please look at the inputfile :
84176527,0047,03,1st
84176527,0047,03,
Is a duplicate record if I want to sort on 1st and 2nd field.

I sorted the issue with :
cat inputfile | sort -t -k1,2 -u > unq
cat inputfile | sort -t -k1,2 > non-unq
comm -23 non-unq unq > duplicates

MatrixMadhan, Jean-Pierre : Thanks.

Thanks.

ghostdog74 · #6 (**permalink**) 05-17-2007

Code:

awk -F"," '{ line[$1.$2] = $0
             arr[$1.$2]++
           }
END{     for (i in arr) {
            if ( arr[i] > 1 ){
	       print line[i] > "duplicates"
	    }
	 } 
 }' file

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
removing duplicates and sort -k	orahi001	UNIX for Dummies Questions & Answers	3	01-25-2008 09:59 AM
Removing duplicates [sort , uniq]	sharatz83	Shell Programming and Scripting	4	07-14-2006 05:12 PM
sort and uniq in perl	reggiej	Shell Programming and Scripting	4	05-18-2006 10:46 PM
Help with Last,uniq, sort and cut	jay1228	UNIX for Dummies Questions & Answers	1	02-16-2005 01:33 AM
sort/uniq	jimmyflip	UNIX for Dummies Questions & Answers	3	10-17-2002 05:09 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by