Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

script to remove duplicates per line

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers script to remove duplicates per line
# 1  
Old 09-25-2012
script to remove duplicates per line
Hello experts!

I'd like a way to remove duplicates per line. Strings are enclosed in brackets, and I would prefer to maintain the order of the file:

example input
Code:
(56)(63)
(56)(70)(56)(70)(24)
(25)(78)
(12)(33)(12)
(10)
(10)

desired output
Code:
(56)(63)
(56)(70)(24)
(25)(78)
(12)(33)
(10)
(10)

Your help is sincerely appreciated...
Thanks
# 2  
Old 09-26-2012
See if the below code will work for you:
Code:
$ cat t
(56)(63)
(56)(70)(56)(70)(24)
(25)(78)
(12)(33)(12)
(10)
(10)

$ while read l
> do
>   echo "$l" | sed -e 's/)(/)\n(/g' -e 's/)$/)\n/' | awk '!x[$0]++' >> t2
> done <t

$ awk  'BEGIN { RS = ""; OFS = ""} {$1 = $1; print }' t2
(56)(63)
(56)(70)(24)
(25)(78)
(12)(33)
(10)
(10)

This User Gave Thanks to spacebar For This Post:
torchij
# 3  
Old 09-26-2012
The following will doing it without creating an intermediate file:
Code:
awk -F ")" '{
        for(i=NF;i>0;i--)
                for(j=1;j < i;j++)
                        if($i == $j) {
                                # get rid of "(<digit>s" from last duplicate
                                $i=""
                                # change "))" to ")" to finish duplicate removal
                                sub(/[)][)]/,")")
                                break
                        }
        print
}' OFS=")" input

This will at least work with ksh, bash, or sh as your shell. It won't work with csh.
Last edited by Don Cragun; 09-26-2012 at 04:55 AM.. Reason: Fix cut and paste error.
This User Gave Thanks to Don Cragun For This Post:
torchij
# 4  
Old 09-26-2012
Code:
sed -e ':a' -e 's/\(([^)]*)\)\(.*\)\1/\1\2/;ta' infile

These 2 Users Gave Thanks to Scrutinizer For This Post:
guruprasadpr torchij
# 5  
Old 09-26-2012
try this..

Code:
awk -F "[()]" '{ for(i=2;i<=NF;i+=2){if(!X[$i]++){printf "("$i")"}}{delete X;print ""}}' file

This User Gave Thanks to pamu For This Post:
torchij
# 6  
Old 09-26-2012
So many solutions! Thanks to everyone.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove duplicates using for loop?

values=(1 2 3 5 4 2 3 1 6 8 3 5 ) #i need the output like this by removing the duplicates 1 2 3 5 4 6 8 #i dont need sorting in my program #plz explain me as simple using for loop #os-ubuntu ,shell=bash (5 Replies)
Discussion started by: Meeran Rizvi
5 Replies

2. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

3. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

4. Shell Programming and Scripting

Help with merge and remove duplicates

Hi all, I need some help to remove duplicates from a file before merging. I have got 2 files: file1 has data in format 4300 23456 4301 2357 the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file file 2 has data in same format but is not in... (10 Replies)
Discussion started by: roy121
10 Replies

5. Shell Programming and Scripting

awk remove first duplicates

Hi All, I have searched many threads for possible close solution. But I was unable to get simlar scenario. I would like to print all duplicate based on 3rd column except the first occurance. Also would like to print if it is single entry(non-duplicate). i/P file 12 NIL ABD LON 11 NIL ABC... (6 Replies)
Discussion started by: sybadm
6 Replies

6. Shell Programming and Scripting

Awk: Remove Duplicates

I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file. Requirement: Can both the awk... (4 Replies)
Discussion started by: siramitsharma
4 Replies

7. Shell Programming and Scripting

delete from line and remove duplicates

My Input.....file1 ABCDE4435 Connected to 107.71.136.122 (SubNetwork=ONRM_RootMo_R SubNetwork=XYVLTN29CRBR99 MeContext=ABCDE4435 ManagedElement=1) ABCDE4478 Connected to 166.208.30.57 (SubNetwork=ONRM_RootMo_R SubNetwork=KLFMTN29CR0R04 MeContext=ABCDE4478 ManagedElement=1) ABCDE4478... (5 Replies)
Discussion started by: pareshkp
5 Replies

8. Shell Programming and Scripting

Script to remove duplicates

Hi I need a script that removes the duplicate records and write it to a new file for example I have a file named test.txt and it looks like abcd.23 abcd.24 abcd.25 qwer.25 qwer.26 qwer.98 I want to pick only $1 and compare with the next record and the output should be abcd.23... (6 Replies)
Discussion started by: antointoronto
6 Replies

9. Shell Programming and Scripting

Shell script to remove duplicates lines in a file

Hi, I am writing a shell script that needs to remove duplicate lines within a file by category. example: section a a c b a section b a b a c I need to remove the duplicates within th category with out removing the duplicates from the 2 different sections (one of the a's in section... (1 Reply)
Discussion started by: RichElks
1 Replies

10. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies
Login or Register to Ask a Question