Remove duplicates from comma separated list


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Remove duplicates from comma separated list
# 1  
Old 11-21-2019
Remove duplicates from comma separated list

Hi,


I have following input file:


Code:
niki      niki
niki1    niki
niki2    niki,niki2
niki3    niki,niki3,niki
niki4    niki4,blabla
niki5    jkjkl
niki6    niki60,niki6

I would like to delete lines with identical matches completely and remove the selfmatches in the other lines.


Expected output:


Code:
niki1    niki
niki2    niki 
niki3    niki,niki
niki4    blabla
niki5    jkjkl
niki6    niki60


Is that somehow possible?


I tried several things with awk but I can not find a way to solve it properly.


I hope someone can help here.


Thanks!

Last edited by Scrutinizer; 11-21-2019 at 05:59 PM..
# 2  
Old 11-21-2019
You could use split() to put $2 in an array and then step thru it like this:

Code:
awk '
{ 
   a=""
   e=split($2, vals, ",")
   for(i=1;i<=e;i++) if(vals[i]!=$1) a=a","vals[i]
   if(a) print $1,substr(a,2)
} ' infile

# 3  
Old 11-22-2019
Try also
Code:
awk '
        {gsub ($1 "(,|$)", _, $2)
         gsub (/,,/, ",", $2)
         gsub (/^,|,$/, _, $2)
         $1 = $1
        }
$2
' file

Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to extract fields from a CSV i.e comma separated where some of the fields having comma as value?

can anyone help me!!!! How to I parse the CSV file file name : abc.csv (csv file) The above file containing data like abv,sfs,,hju,',',jkk wff,fst,,rgr,',',rgr ere,edf,erg,',',rgr,rgr I have a requirement like i have to extract different field and assign them into different... (4 Replies)
Discussion started by: J.Jena
4 Replies

2. Shell Programming and Scripting

Remove duplicates separated by delimiter

First post, been browsing for 3 days and came out with nothing so far. M3 C2 V5 D5 HH:FF A1-A2,A5-A6,A1-A2,A1-4 B4-B6,B2-B4,B4-B6,B1-B2output should be M3 C2 V5 D5 HH:FF A1-A2,A5-A6,A1-A4 B2-B4,B4-B6,B1-B2On col 6 and 7 there are strings in form of Ax-Ax and Bx-Bx respectively. Each string are... (9 Replies)
Discussion started by: enrikS
9 Replies

3. Shell Programming and Scripting

awk to parse comma separated field and removing comma in between number and double quotes

Hi Experts, Please support I have below data in file in comma seperated, but 4th column is containing comma in between numbers, bcz of which when i tried to parse the file the column 6th value(5049641141) is being removed from the file and value(222.82) in column 5 becoming value of column6. ... (3 Replies)
Discussion started by: as7951
3 Replies

4. UNIX for Dummies Questions & Answers

[solved] Comma separated values to space separated

Hi, I have a large number of files which are written as csv (comma-separated values). Does anyone know of simple sed/awk command do achieve this? Thanks! ---------- Post updated at 10:59 AM ---------- Previous update was at 10:54 AM ---------- Guess I asked this too soon. Found the... (0 Replies)
Discussion started by: lost.identity
0 Replies

5. Shell Programming and Scripting

Need Help - comma inside double quote in comma separated csv,

Hello there, I have a comma separated csv , and all the text field is wrapped by double quote. Issue is some text field contain comma as well inside double quote. so it is difficult to process. Input in the csv file is , 1,234,"abc,12,gh","GH234TY",34 I need output like below,... (8 Replies)
Discussion started by: Uttam Maji
8 Replies

6. Shell Programming and Scripting

Output of command in comma separated list

Hi; I have an output of a particular command say $command fstl:r-x ajay:r-x how can i get this in comma separated list, eg: fstl:r-x,ajay:r-x Thnks; (4 Replies)
Discussion started by: ajaypadvi
4 Replies

7. Shell Programming and Scripting

Help parse comma separated list

I have a list of files with the same name, but they have a different date stamp in the name. I can find the first file, but I need to find the second file. I am using this information to create a variable I use later. Here is a example of how I find the first file. "ls -mr... (11 Replies)
Discussion started by: NoMadBanker
11 Replies

8. Shell Programming and Scripting

Need comma separated processing

I have a file like this OUTLN OPEN Y SCOTT OPEN N USER4 OPEN Y DBSNMP EXPIRED & LOCKED N ... (4 Replies)
Discussion started by: ilugopal
4 Replies

9. UNIX for Dummies Questions & Answers

Remove whitespaces between comma separated fields from file

Hello all, I am a unix dummy. I am trying to remove spaces between fields. I have the file in the following format 12332432, 2345 , asdfsdf ,100216 , 9999999 12332431, 2341 , asdfsd2 ,100213 , 9999999 &... (2 Replies)
Discussion started by: nitinbjoshi
2 Replies
Login or Register to Ask a Question