Home Man
Search
Today's Posts
Register

BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

finding duplicates in csv based on key columns

Tags
shell scripts

Login to Reply

 
Thread Tools Search this Thread
# 1  
Old 11-24-2011
finding duplicates in csv based on key columns

Hi team,

I have 20 columns csv files. i want to find the duplicates in that file based on the column1 column10 column4 column6 coulnn8 coulunm2 . if those columns have same values . then it should be a duplicate record.

can one help me on finding the duplicates,

Thanks in advance.

i sorted the columns first based on the key and view the data but. it won't show properly.

thanks,
Baski
# 2  
Old 11-24-2011
Give some input content and expected output ..
# 3  
Old 11-24-2011
Try this:

Quote:

'inputFile.csv'
------------
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
11,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,6,7,8,9,40,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20
1,2,3,4,5,66,7,8,9,10,11,12,13,14,15,16,17,18,19,20


awk -F, '!dup[$1,$10,$4,$6,$8,$2]++' inputFile.csv
Login to Reply

« Previous Thread | Next Thread »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Sort and remove duplicates in directory based on first 5 columns: gnnsprapa UNIX for Beginners Questions & Answers 4 02-09-2018 05:50 PM
Removing duplicates from delimited file based on 2 columns kevinprood Shell Programming and Scripting 2 08-13-2014 04:37 AM
UNIX scripting for finding duplicates and null records in pk columns praveenraj.1991 Shell Programming and Scripting 5 05-11-2014 04:20 AM
Finding duplicates then copying, almost there, maybe? Rhinoskin UNIX for Dummies Questions & Answers 2 12-16-2011 12:45 AM
Help finding non duplicates chipblah84 Shell Programming and Scripting 6 06-03-2011 03:10 AM
Search based on 1,2,4,5 columns and remove duplicates in the same file. onesuri Shell Programming and Scripting 2 10-25-2010 05:00 AM
Remove duplicates based on the two key columns kmsekhar Shell Programming and Scripting 7 10-21-2010 11:12 AM
Finding duplicates from positioned substring across lines gapprasath Shell Programming and Scripting 2 12-24-2008 04:43 AM
finding duplicates in columns and removing lines totus Shell Programming and Scripting 17 11-29-2008 10:27 AM
finding duplicates with perl dangral Shell Programming and Scripting 3 01-28-2003 11:50 AM


All times are GMT -4. The time now is 01:19 AM.

Unix & Linux Forums Content Copyright©1993-2018. All Rights Reserved.
UNIX.COM Login
Username:
Password:  
Show Password