Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

How to remove duplicates without sorting

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers How to remove duplicates without sorting
# 1  
Old 01-16-2008
How to remove duplicates without sorting
Hello,


I can remove duplicate entries in a file by:

sort File1 | uniq > File2

but how can I remove duplicates without sorting the file?

I tried cat File1 | uniq > File2 but it doesn't work



thanks
# 2  
Old 01-16-2008
Quote:
Originally Posted by orahi001
Hello,


I can remove duplicate entries in a file by:

sort File1 | uniq > File2

but how can I remove duplicates without sorting the file?

I tried cat File1 | uniq > File2 but it doesn't work



thanks
Many ways using awk or perl. Search this site using the duplicates keyword.
# 3  
Old 01-16-2008
Before my post I found on an older post

awk ' !x[$0]++' Infile > outFile


but it was giving syntax errors when exectued.

Then I changed awk to nawk and it worked.



thanks
# 4  
Old 01-17-2008
Quote:
sort File1 | uniq > File2
You don't need a sort and uniq for removing duplicates

same can be accomplished by "-u" option

Code:
sort -u filename > new.filename

# 5  
Old 01-17-2008
Quote:
Originally Posted by orahi001
Before my post I found on an older post

awk ' !x[$0]++' Infile > outFile


but it was giving syntax errors when exectued.

Then I changed awk to nawk and it worked.



thanks

try to use nawk or /usr/xpg4/bin/awk
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

2. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies

3. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

4. Shell Programming and Scripting

awk remove first duplicates

Hi All, I have searched many threads for possible close solution. But I was unable to get simlar scenario. I would like to print all duplicate based on 3rd column except the first occurance. Also would like to print if it is single entry(non-duplicate). i/P file 12 NIL ABD LON 11 NIL ABC... (6 Replies)
Discussion started by: sybadm
6 Replies

5. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

6. Shell Programming and Scripting

Perl, sorting and eliminating duplicates

Hi guys! I'm trying to eliminate some duplicates from a file but I'm like this :wall: !!! My file looks like this: ID_1 0.02 ID_2 2.4e-2 ID_2 4.3.e-9 ID_3 0.003 ID_4 0.2 ID_4 0.05 ID_5 1.2e-3 What I need is to eliminate all the duplicates considering the first column (in this... (6 Replies)
Discussion started by: gabrysfe
6 Replies

7. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks (6 Replies)
Discussion started by: dvah
6 Replies

8. Shell Programming and Scripting

Script to remove duplicates

Hi I need a script that removes the duplicate records and write it to a new file for example I have a file named test.txt and it looks like abcd.23 abcd.24 abcd.25 qwer.25 qwer.26 qwer.98 I want to pick only $1 and compare with the next record and the output should be abcd.23... (6 Replies)
Discussion started by: antointoronto
6 Replies

9. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

10. Shell Programming and Scripting

fastest way to remove duplicates.

I have searched the FAQ - by using sort, duplicates, etc.... but I didn't get any articles or results on it. Currently, I am using: sort -u file1 > file2 to remove duplicates. For a file size of 1giga byte approx. time taken to remove duplicates is 1hr 21 mins. Is there any other faster way... (15 Replies)
Discussion started by: radhika
15 Replies
Login or Register to Ask a Question