Unix/Linux Go Back    


UNIX for Beginners Questions & Answers If you're not sure where to post a Unix or Linux question, post it here. All unix and Linux beginners welcome in this forum!

How can I remove partial duplicates and manipulate text?

UNIX for Beginners Questions & Answers


Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 2 Weeks Ago   -   Original Discussion by tara123
tara123's Unix or Linux Image
tara123 tara123 is offline
Registered User
 
Join Date: Jan 2013
Last Activity: 30 December 2017, 11:53 PM EST
Posts: 41
Thanks: 2
Thanked 0 Times in 0 Posts
How can I remove partial duplicates and manipulate text?

Hello,

How can I remove partial duplicates and manipulate text in bash using either awk, grep or sed? Thanks.


Input:



Code:
ted,"foo,bar,zoo"
john-son,"foot,ben,zoo"
bob,"bar,foot"

Expected Output:



Code:
foo,ted
bar,ted
zoo,ted
foot,john-son
ben,john-son

Sponsored Links
    #2  
Old Unix and Linux 2 Weeks Ago   -   Original Discussion by tara123
Scrutinizer's Unix or Linux Image
Scrutinizer Scrutinizer is offline Forum Staff  
Moderator
 
Join Date: Nov 2008
Last Activity: 17 January 2018, 12:17 PM EST
Location: Amsterdam
Posts: 11,704
Thanks: 530
Thanked 3,398 Times in 2,994 Posts
What have you tried so far?
Sponsored Links
    #3  
Old Unix and Linux 2 Weeks Ago   -   Original Discussion by tara123
tara123's Unix or Linux Image
tara123 tara123 is offline
Registered User
 
Join Date: Jan 2013
Last Activity: 30 December 2017, 11:53 PM EST
Posts: 41
Thanks: 2
Thanked 0 Times in 0 Posts
this did not work.



Code:
perl -lpe 's/\s\K\S+/join ",", grep {!$seen{$_}++} split ",", $&/e'

    #4  
Old Unix and Linux 2 Weeks Ago   -   Original Discussion by tara123
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 17 January 2018, 1:40 PM EST
Location: San Jose, CA, USA
Posts: 10,913
Thanks: 608
Thanked 3,817 Times in 3,261 Posts
It is interesting that you want code written in awk, grep, or sed but show us non-working perl code. Linux

You might be able to use something like:


Code:
awk -F, -v OFS=, '
{	gsub(/"/, "")
	for(i = 2; i <= NF; i++)
		if(!($i in seen)) {
			seen[$i]
			print $i, $1
		}
}' file

which, if file contains your sample input, produces the output you said you wanted.

If you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.
Sponsored Links
    #5  
Old Unix and Linux 2 Weeks Ago   -   Original Discussion by tara123
tara123's Unix or Linux Image
tara123 tara123 is offline
Registered User
 
Join Date: Jan 2013
Last Activity: 30 December 2017, 11:53 PM EST
Posts: 41
Thanks: 2
Thanked 0 Times in 0 Posts
Thank you very much. It worked.
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Script to compare partial filenames in two folders and delete duplicates shaky Shell Programming and Scripting 2 01-12-2017 11:33 AM
Match partial text cmccabe Shell Programming and Scripting 7 03-10-2015 01:49 PM
Manipulate the text file in UNIX siva.pitchai Shell Programming and Scripting 6 11-27-2014 03:40 AM
sed to remove partial text in one line only kumars1331@gmai Shell Programming and Scripting 3 10-12-2014 09:54 PM
Remove the partial duplicates by checking the length of a field asyed Shell Programming and Scripting 3 09-23-2011 10:33 AM



All times are GMT -4. The time now is 04:16 PM.