Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Script to remove duplicates

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to remove duplicates
# 1  
Old 04-28-2010
Script to remove duplicates
Hi

I need a script that removes the duplicate records and write it to a new file
for example I have a file named test.txt and it looks like

abcd.23
abcd.24
abcd.25
qwer.25
qwer.26
qwer.98

I want to pick only $1 and compare with the next record and the output should be
abcd.23
qwer.25

can some one help?
# 2  
Old 04-28-2010
Code:
#!/bin/sh

PREV_A=""
while IFS="." read A B
do
        [ "${PREV_A}" == "$A" ] || echo "$A.$B"
        PREV_A="$A"
done < file

# 3  
Old 04-28-2010
Try this:
Code:
awk -F\. '!a[$1]++' file

BTW: per forum rules, and the benefit of all users, please search the network and the forums before posting a question.

You can easily search the forums using our internal Google search engine or our advanced internal search engine. You can also search our huge UNIX and Linux database by user generated tags or search the database for unanswered questions and provide an answer.

Thank you.

The UNIX and Linux Forums
# 4  
Old 04-28-2010
Code:
$ cat file
abcd.23
abcd.24
abcd.25
qwer.25
qwer.26
qwer.98
$ sort -u -t"." -k1,1 file
abcd.23
qwer.25

# 5  
Old 04-28-2010
a bash one
Code:
IFS='.'
while read A B
do
	[ "$A" = "$C" ] || echo "$A.$B"
	C=$A
done < file

Or, assuming there are always 4 characters to compare
Code:
uniq -w4 file

# 6  
Old 04-28-2010
test.txt (input)
Code:
abcd.23
abcd.24
abcd.25
qwer.25
qwer.26
qwer.98
wxyz.48
wxyz.55
wxyz.70

Code:
$ touch ntest.txt
$ for i in `cat test.txt | cut -d'.' -f1 | uniq`
do
grep -m1 $i test.txt >> ntest.txt
done
$

ntest.txt (output)
Code:
abcd.23
qwer.25
wxyz.48

Warning: This code will work only if there are no whitespaces in string before "."
# 7  
Old 04-29-2010
thanks guys. it worked
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove duplicates using for loop?

values=(1 2 3 5 4 2 3 1 6 8 3 5 ) #i need the output like this by removing the duplicates 1 2 3 5 4 6 8 #i dont need sorting in my program #plz explain me as simple using for loop #os-ubuntu ,shell=bash (5 Replies)
Discussion started by: Meeran Rizvi
5 Replies

2. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

3. Shell Programming and Scripting

Remove top 3 duplicates

hello , I have a requirement with input in below format abc 123 xyz bcd 365 kii abc 987 876 cdf 987 uii abc 456 yuu bcd 654 rrr Expecting Output abc 456 yuu bcd 654 rrr cdf 987 uii (1 Reply)
Discussion started by: Tomlight
1 Replies

4. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

5. UNIX for Dummies Questions & Answers

script to remove duplicates per line

Hello experts! I'd like a way to remove duplicates per line. Strings are enclosed in brackets, and I would prefer to maintain the order of the file: example input (56)(63) (56)(70)(56)(70)(24) (25)(78) (12)(33)(12) (10) (10) desired output (56)(63) (56)(70)(24) (25)(78)... (5 Replies)
Discussion started by: torchij
5 Replies

6. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

7. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks (6 Replies)
Discussion started by: dvah
6 Replies

8. Shell Programming and Scripting

Remove duplicates from a file

Hi, I need to remove duplicates from a file. The file will be like this 0003 10101 20100120 abcdefghi 0003 10101 20100121 abcdefghi 0003 10101 20100122 abcdefghi 0003 10102 20100120 abcdefghi 0003 10103 20100120 abcdefghi 0003 10103 20100121 abcdefghi Here if the first colum and... (6 Replies)
Discussion started by: gpaulose
6 Replies

9. Shell Programming and Scripting

Shell script to remove duplicates lines in a file

Hi, I am writing a shell script that needs to remove duplicate lines within a file by category. example: section a a c b a section b a b a c I need to remove the duplicates within th category with out removing the duplicates from the 2 different sections (one of the a's in section... (1 Reply)
Discussion started by: RichElks
1 Replies

10. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies
Login or Register to Ask a Question