Remove duplicate phrase

Login or Register to Ask a Question and Join Our Community

Remove duplicate phrase

Tags

Login to Discuss or Reply to this Discussion in Our Community

Top Forums Shell Programming and Scripting Remove duplicate phrase

10-16-2006

Registered User

10, 0

Join Date: Aug 2005

Last Activity: 30 October 2014, 5:02 PM EDT

Posts: 10

Thanks Given: 0

Thanked 0 Times in 0 Posts

Remove duplicate phrase

I have a value
VAL=XXXX_20061001_00 XXXX_20061001_03 XXXX_20061001_00 XXXX_20061002_00 XXXX_20061002_03 XXXX_20061003_00

The VAL has duplicates like XXXX_20061001_00 repeated twice.

How to make the VAL with unique values.

Thanks in advance.

kesu2k

View Public Profile for kesu2k

Find all posts by kesu2k

10-16-2006

Registered User

2,205, 181

Join Date: Mar 2006

Last Activity: 8 May 2020, 5:01 AM EDT

Location: Bangalore,India

Posts: 2,205

Thanks Given: 31

Thanked 181 Times in 171 Posts

Code:

echo $val | 
awk ' 
{
   for(i=1;i<=NF;++i)  arr[$i]; 
   for(key in arr) printf("%s ",key)
}'

anbu23

View Public Profile for anbu23

Find all posts by anbu23

10-16-2006

Registered User

10, 0

Join Date: Aug 2005

Last Activity: 30 October 2014, 5:02 PM EDT

Posts: 10

Thanks Given: 0

Thanked 0 Times in 0 Posts

Thanks it really worked.

kesu2k

View Public Profile for kesu2k

Find all posts by kesu2k

10-16-2006

Registered User

2,669, 20

Join Date: Sep 2006

Last Activity: 28 January 2015, 8:30 AM EST

Posts: 2,669

Thanks Given: 0

Thanked 20 Times in 20 Posts

Python alternative:

Code:

s="VAL=XXXX_20061001_00 XXXX_20061001_03 XXXX_20061001_00 XXXX_20061002_00 XXXX_20061002_03 XXXX_20061003_00"
for i in sorted(set(s.split("=")[-1].split())):
     print i,

output:

Code:

XXXX_20061002_00 XXXX_20061002_03 XXXX_20061001_00 XXXX_20061003_00 XXXX_20061001_03

Last edited by ghostdog74; 10-17-2006 at 08:18 PM..

ghostdog74

View Public Profile for ghostdog74

Find all posts by ghostdog74

10-17-2006

Registered User

10, 0

Join Date: Aug 2005

Last Activity: 30 October 2014, 5:02 PM EDT

Posts: 10

Thanks Given: 0

Thanked 0 Times in 0 Posts

Thanks it worked ,can it be sorted to get the output like this

XXXX_20061001_00 XXXX_20061001_03 XXXX_20061002_00 XXXX_20061002_03 XXXX_20061003_00

Thanks once again

kesu2k

View Public Profile for kesu2k

Find all posts by kesu2k

10-17-2006

Registered User

2,205, 181

Join Date: Mar 2006

Last Activity: 8 May 2020, 5:01 AM EDT

Location: Bangalore,India

Posts: 2,205

Thanks Given: 31

Thanked 181 Times in 171 Posts

Quote:

Originally Posted by kesu2k

Thanks it worked ,can it be sorted to get the output like this

XXXX_20061001_00 XXXX_20061001_03 XXXX_20061002_00 XXXX_20061002_03 XXXX_20061003_00

Thanks once again Smilie

Smilie

Code:

echo $VAL | tr ' ' '\n' | sort -u |sort -t"_" -n -k 2 | tr '\n' ' '

anbu23

View Public Profile for anbu23

Find all posts by anbu23

10-17-2006

Registered User

10, 0

Join Date: Aug 2005

Last Activity: 30 October 2014, 5:02 PM EDT

Posts: 10

Thanks Given: 0

Thanked 0 Times in 0 Posts

Thanks!! thats Gr8t

kesu2k

View Public Profile for kesu2k

Find all posts by kesu2k

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to find a phrase and pull all lines that follow until the phrase occurs again?

I want to burst a report by using the page number value in the report header. Each section starts with *PAGE NO:* 1 Each section might have several pages, but the next section always starts back at 1. So I want to find the "*PAGE NO:* 1" value and pull all lines that follow until "*PAGE NO:* 1"...

2. UNIX for Dummies Questions & Answers

Remove duplicate

Hi, How can I replace || with space and then remove duplicate from following text? T111||T222||T444||T222||T555 Thanks in advance

3. Shell Programming and Scripting

Remove duplicate

Hi , I have a pipe seperated file repo.psv where i need to remove duplicates based on the 1st column only. Can anyone help with a Unix script ? Input: 15277105||Common Stick|ESHR||Common Stock|CYRO AB 15277105||Common Stick|ESHR||Common Stock|CYRO AB 16111278||Common Stick|ESHR||Common...

4. Shell Programming and Scripting

How to Remove duplicate value from file?

if different branch code is available for same BIC code and one of the branch code is XXX.only one row will be stored and with branch code as XXX .rest of the rows for the BIC code will not be stored. for example if $7 is BIC code and $8 is branch code INPUT file are following...

5. Shell Programming and Scripting

How to remove duplicate ID's?

HI I have file contains 1000'f of duplicate id's with (upper and lower first character) as below i/p: a411532A411532a508661A508661c411532C411532 Requirement: But i need to ignore lowercase id's and need only below id's o/p: A411532 A508661 C411532

6. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few...

7. Shell Programming and Scripting

remove duplicate

Hi, I am tryung to use shell or perl to remove duplicate characters for example , if I have " I love google" it will become I love ggle" or even "I loveggle" if removing duplicate white space Thanks CC

8. Shell Programming and Scripting

Remove duplicate

Hi all, I have a text file fileA.txt DXRV|02/28/2006 11:36:49.049|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:37:06.404|SAC||||CDxAcct=6070970034 DXRV|02/28/2006 11:37:25.740|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:38:32.633|SAC||||CDxAcct=6070970034 DXRV|02/28/2006...

9. Shell Programming and Scripting

Remove duplicate ???

Hi all, I have a out.log file CARR|02/26/2006 10:58:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:11:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:18:30.107|CDxAcct=7659579782 CARR|02/26/2006 11:28:30.107|CDxAcct=9534922327 CARR|02/26/2006 11:38:30.107|CDxAcct=9534922327 CARR|02/26/2006...

10. Shell Programming and Scripting

remove duplicate

i have a text its contain many record, but its written in one line, i want to remove from that line the duplicate record, not record have fixed width ex: width = 4 inputfile test.txt =abc cdf abc abc cdf fgh fgh abc abc i want the outputfile =abc cdf fgh only those records can any one help...

Login or Register to Ask a Question