Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Remove duplicates from comma separated list Post 303041350 by niki0211 on Thursday 21st of November 2019 04:13:14 PM
Old 11-21-2019
Remove duplicates from comma separated list

Hi,


I have following input file:


Code:
niki      niki
niki1    niki
niki2    niki,niki2
niki3    niki,niki3,niki
niki4    niki4,blabla
niki5    jkjkl
niki6    niki60,niki6

I would like to delete lines with identical matches completely and remove the selfmatches in the other lines.


Expected output:


Code:
niki1    niki
niki2    niki 
niki3    niki,niki
niki4    blabla
niki5    jkjkl
niki6    niki60


Is that somehow possible?


I tried several things with awk but I can not find a way to solve it properly.


I hope someone can help here.


Thanks!

Last edited by Scrutinizer; 11-21-2019 at 05:59 PM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remove whitespaces between comma separated fields from file

Hello all, I am a unix dummy. I am trying to remove spaces between fields. I have the file in the following format 12332432, 2345 , asdfsdf ,100216 , 9999999 12332431, 2341 , asdfsd2 ,100213 , 9999999 &... (2 Replies)
Discussion started by: nitinbjoshi
2 Replies

2. Shell Programming and Scripting

Need comma separated processing

I have a file like this OUTLN OPEN Y SCOTT OPEN N USER4 OPEN Y DBSNMP EXPIRED & LOCKED N ... (4 Replies)
Discussion started by: ilugopal
4 Replies

3. Shell Programming and Scripting

Help parse comma separated list

I have a list of files with the same name, but they have a different date stamp in the name. I can find the first file, but I need to find the second file. I am using this information to create a variable I use later. Here is a example of how I find the first file. "ls -mr... (11 Replies)
Discussion started by: NoMadBanker
11 Replies

4. Shell Programming and Scripting

Output of command in comma separated list

Hi; I have an output of a particular command say $command fstl:r-x ajay:r-x how can i get this in comma separated list, eg: fstl:r-x,ajay:r-x Thnks; (4 Replies)
Discussion started by: ajaypadvi
4 Replies

5. Shell Programming and Scripting

Need Help - comma inside double quote in comma separated csv,

Hello there, I have a comma separated csv , and all the text field is wrapped by double quote. Issue is some text field contain comma as well inside double quote. so it is difficult to process. Input in the csv file is , 1,234,"abc,12,gh","GH234TY",34 I need output like below,... (8 Replies)
Discussion started by: Uttam Maji
8 Replies

6. UNIX for Dummies Questions & Answers

[solved] Comma separated values to space separated

Hi, I have a large number of files which are written as csv (comma-separated values). Does anyone know of simple sed/awk command do achieve this? Thanks! ---------- Post updated at 10:59 AM ---------- Previous update was at 10:54 AM ---------- Guess I asked this too soon. Found the... (0 Replies)
Discussion started by: lost.identity
0 Replies

7. Shell Programming and Scripting

awk to parse comma separated field and removing comma in between number and double quotes

Hi Experts, Please support I have below data in file in comma seperated, but 4th column is containing comma in between numbers, bcz of which when i tried to parse the file the column 6th value(5049641141) is being removed from the file and value(222.82) in column 5 becoming value of column6. ... (3 Replies)
Discussion started by: as7951
3 Replies

8. Shell Programming and Scripting

Remove duplicates separated by delimiter

First post, been browsing for 3 days and came out with nothing so far. M3 C2 V5 D5 HH:FF A1-A2,A5-A6,A1-A2,A1-4 B4-B6,B2-B4,B4-B6,B1-B2output should be M3 C2 V5 D5 HH:FF A1-A2,A5-A6,A1-A4 B2-B4,B4-B6,B1-B2On col 6 and 7 there are strings in form of Ax-Ax and Bx-Bx respectively. Each string are... (9 Replies)
Discussion started by: enrikS
9 Replies

9. UNIX for Beginners Questions & Answers

How to extract fields from a CSV i.e comma separated where some of the fields having comma as value?

can anyone help me!!!! How to I parse the CSV file file name : abc.csv (csv file) The above file containing data like abv,sfs,,hju,',',jkk wff,fst,,rgr,',',rgr ere,edf,erg,',',rgr,rgr I have a requirement like i have to extract different field and assign them into different... (4 Replies)
Discussion started by: J.Jena
4 Replies
IO::Async::Protocol::LineStream(3pm)			User Contributed Perl Documentation		      IO::Async::Protocol::LineStream(3pm)

NAME
"IO::Async::Protocol::LineStream" - stream-based protocols using lines of text SYNOPSIS
Most likely this class will be subclassed to implement a particular network protocol. package Net::Async::HelloWorld; use strict; use warnings; use base qw( IO::Async::Protocol::LineStream ); sub on_read_line { my $self = shift; my ( $line ) = @_; if( $line =~ m/^HELLO (.*)/ ) { my $name = $1; $self->invoke_event( on_hello => $name ); } } sub send_hello { my $self = shift; my ( $name ) = @_; $self->write_line( "HELLO $name" ); } This small example elides such details as error handling, which a real protocol implementation would be likely to contain. DESCRIPTION
EVENTS
The following events are invoked, either using subclass methods or CODE references in parameters: on_read_line $line Invoked when a new complete line of input is received. PARAMETERS
The following named parameters may be passed to "new" or "configure": on_read_line => CODE CODE reference for the "on_read_line" event. METHODS
$lineprotocol->write_line( $text ) Writes a line of text to the transport stream. The text will have the end-of-line marker appended to it; $text should not end with it. AUTHOR
Paul Evans <leonerd@leonerd.org.uk> perl v5.14.2 2012-10-24 IO::Async::Protocol::LineStream(3pm)
All times are GMT -4. The time now is 02:33 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy