Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

finding duplicates in csv based on key columns

Shell Programming and Scripting


Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 11-24-2011   -   Original Discussion by baskivs
baskivs baskivs is offline
Registered User
 
Join Date: Jun 2011
Last Activity: 4 April 2012, 2:36 AM EDT
Posts: 27
Thanks: 3
Thanked 0 Times in 0 Posts
finding duplicates in csv based on key columns

Hi team,

I have 20 columns csv files. i want to find the duplicates in that file based on the column1 column10 column4 column6 coulnn8 coulunm2 . if those columns have same values . then it should be a duplicate record.

can one help me on finding the duplicates,

Thanks in advance.

i sorted the columns first based on the key and view the data but. it won't show properly.

thanks,
Baski
Sponsored Links
    #2  
Old Unix and Linux 11-24-2011   -   Original Discussion by baskivs
jayan_jay's Unix or Linux Image
jayan_jay jayan_jay is offline Forum Advisor  
Forum Advisor
 
Join Date: Jul 2008
Last Activity: 9 March 2016, 9:36 AM EST
Posts: 833
Thanks: 9
Thanked 185 Times in 176 Posts
Give some input content and expected output ..
Sponsored Links
    #3  
Old Unix and Linux 11-24-2011   -   Original Discussion by baskivs
Sheel Sheel is offline
Registered User
 
Join Date: May 2010
Last Activity: 9 July 2012, 6:56 AM EDT
Location: Bangalore
Posts: 60
Thanks: 1
Thanked 1 Time in 1 Post
Try this:

Quote:

'inputFile.csv'
------------
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
11,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,6,7,8,9,40,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,66,7,8,9,10,11,12,13,14,15,16,17,18,19,20


awk -F, '!dup[$1,$10,$4,$6,$8,$2]++' inputFile.csv
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Help finding non duplicates chipblah84 Shell Programming and Scripting 6 06-03-2011 04:10 AM
Search based on 1,2,4,5 columns and remove duplicates in the same file. onesuri Shell Programming and Scripting 2 10-25-2010 06:00 AM
Remove duplicates based on the two key columns kmsekhar Shell Programming and Scripting 7 10-21-2010 12:12 PM
finding duplicates in columns and removing lines totus Shell Programming and Scripting 17 11-29-2008 11:27 AM
finding duplicates with perl dangral Shell Programming and Scripting 3 01-28-2003 12:50 PM



All times are GMT -4. The time now is 05:10 AM.