Hello experts,

I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter


sort -u -k 2,2 File.csv > Output.csv


File.csv
File Name|Document Name|Document Title|Organization
Word Doc 1.doc|Word Document|Sample Doc|Org 1
Exl Doc 1.xls|Excel Sheet|Sample Sheet|Org 2
Pdf File 1.pdf|Pdf|Sample pdf|Org3
Exl Sheet 2.xls|Excel Sheet|Test Spreadsheet|Org 2



I want Output.csv to remove the 2nd Excell Sheet line
Output.csv
File Name|Document Name|Document Title|Organization
Word Doc 1.doc|Word Document|Sample Doc|Org 1
Exl Doc 1.xls|Excel Sheet|Sample Sheet|Org 2
Pdf File 1.pdf|Pdf|Sample pdf|Org3


I believe the -k option uses spaces to determine the start and end fields

My file seperator is a '|' so I want to remove the line with the duplicate Document Name (2nd column).

Can this be done using the -k option of sort or is there another way to perform this task?


thanks

try and "man sort" to see what the options for sort are on you system.

you need to specify the delimiter:

The -t option worked great

I also got it working using nawk

nawk -F'|' '!a[$3]++'


thanks