×
UNIX.COM Login
Username:
Password:  
Show Password






👤


Shell Programming and Scripting

BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

finding duplicates in csv based on key columns

👤 Login to reply

 
Thread Tools Search this Thread Display Modes
    #1  
Old 11-24-2011
baskivs baskivs is offline
Registered User
 
Join Date: Jun 2011
Last Activity: 4 April 2012, 2:36 AM EDT
Posts: 27
Thanks: 3
Thanked 0 Times in 0 Posts
finding duplicates in csv based on key columns

Hi team,

I have 20 columns csv files. i want to find the duplicates in that file based on the column1 column10 column4 column6 coulnn8 coulunm2 . if those columns have same values . then it should be a duplicate record.

can one help me on finding the duplicates,

Thanks in advance.

i sorted the columns first based on the key and view the data but. it won't show properly.

thanks,
Baski
Sponsored Links
    #2  
Old 11-24-2011
jayan_jay's Unix or Linux Image
jayan_jay jayan_jay is offline Forum Advisor  
Forum Advisor
 
Join Date: Jul 2008
Last Activity: 9 March 2016, 9:36 AM EST
Posts: 833
Thanks: 9
Thanked 186 Times in 177 Posts
Give some input content and expected output ..
Sponsored Links
    #3  
Old 11-24-2011
Sheel Sheel is offline
Registered User
 
Join Date: May 2010
Last Activity: 9 July 2012, 6:56 AM EDT
Location: Bangalore
Posts: 60
Thanks: 1
Thanked 1 Time in 1 Post
Try this:

Quote:

'inputFile.csv'
------------
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
11,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,6,7,8,9,40,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,66,7,8,9,10,11,12,13,14,15,16,17,18,19,20


awk -F, '!dup[$1,$10,$4,$6,$8,$2]++' inputFile.csv
Sponsored Links
👤 Login to reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Help finding non duplicates chipblah84 Shell Programming and Scripting 6 06-03-2011 03:10 AM
Search based on 1,2,4,5 columns and remove duplicates in the same file. onesuri Shell Programming and Scripting 2 10-25-2010 05:00 AM
Remove duplicates based on the two key columns kmsekhar Shell Programming and Scripting 7 10-21-2010 11:12 AM
finding duplicates in columns and removing lines totus Shell Programming and Scripting 17 11-29-2008 10:27 AM
finding duplicates with perl dangral Shell Programming and Scripting 3 01-28-2003 11:50 AM



All times are GMT -4. The time now is 12:05 AM.

Unix & Linux Forums Content Copyright©1993-2018. All Rights Reserved.