Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

uniq command

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users uniq command
# 1  
Old 08-10-2011
uniq command
at which situation uniq command fails.

How to delete/remove duplicate lines in an unix file
# 2  
Old 08-10-2011
What is your input like? Is order important? uniq requires lines in sorted order.

If your input data is small enough to be held in main memory, this may do:

Code:
awk '{ if(!arr[$0]) { arr[$0]=1; print; } }' < data

But if your input data is larger than a few handfuls of megs, this is inefficient, you may have to sort it with
Code:
sort -u

which will sort it and remove duplicates.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Trying to find the distinct lines using uniq command

Platform :Oracle Linux 6.4 Shell : bash The below file has 7 lines , some of them are duplicates. There are only 3 distinct lines. But why is the uniq command still showing 7 ? I just want the distinct lines to be returned. $ cat test.txt SELECT FC.COORD_SET_ID FROM OM_ORDER_FLOW F, -... (2 Replies)
Discussion started by: kraljic
2 Replies

2. Shell Programming and Scripting

cat and uniq command

Hello i have tried to retrieve a distinct list of data from a file with a command combination but the uniq command line is not filtering the fields repeated. Can someone help me ? here is the file input 3837734|LAUNCH TEST01 3837735|LAUNCH TEST01 3837736|LAUNCH TEST01 3837737|LAUNCH... (4 Replies)
Discussion started by: ade05fr
4 Replies

3. Shell Programming and Scripting

Addition as part of uniq command

Dear Friends, I want to know if it is possible to perform addition as part of uniq command. Input: 213.64.56.208 1 213.64.56.208 2 213.64.56.208 3 213.46.27.204 10 213.46.27.204 20 213.46.27.204 30 The above input should be converted as follows 213.64.56.208 6 213.46.27.204 60 ... (4 Replies)
Discussion started by: tamil.pamaran
4 Replies

4. UNIX for Dummies Questions & Answers

Re: How To Use UNIQ UNIX Command On single Column

Hi , Can You Please let Know How use unix uniq command on a single column for deleting records from file with Below Structure.Pipe Delimter File . Source Name | Account_Id A | 101 B... (2 Replies)
Discussion started by: anudeepkumar123
2 Replies

5. Shell Programming and Scripting

Uniq Command?

Hello, I have a file with a list of car makes and specific information for each make. An example is: @Audi:Warranty @Audi:Pricing @Audi:Colors @Acura:Warranty @Acura:Pricing @Acura:Colors and so on through a bunch of makes. I need to make a list in a selection box of just one name of... (12 Replies)
Discussion started by: macbb1117
12 Replies

6. UNIX for Dummies Questions & Answers

Several file comparison not uniq or comm command

When comparing several files is there a way to find values unique to each file? File1 a b c d File2 a b t File 3 a (3 Replies)
Discussion started by: dr_sabz
3 Replies

7. UNIX for Dummies Questions & Answers

Uniq command behaving odd

My file has k s j v l k a s f l k s a d f j l a s (3 Replies)
Discussion started by: phoenix_nebula
3 Replies

8. Shell Programming and Scripting

Help with uniq command

I call.... cat 1.txt | uniq -c Sample of whats in 1.txt vmstat cd exit w cd cd cd newgrp xinit f cd cd cd rlogin (2 Replies)
Discussion started by: Bandit390
2 Replies

9. UNIX for Dummies Questions & Answers

Difference between plain "uniq" and "uniq -u"

Dear all, It's not entirely clear to me from manpage the difference between them. Why we still need "-u" flag? - monkfan (3 Replies)
Discussion started by: monkfan
3 Replies

10. UNIX for Dummies Questions & Answers

uniq command???

HI, Please tell the usage of uniq command in UNIX with example Thanks (2 Replies)
Discussion started by: skyineyes
2 Replies
Login or Register to Ask a Question