Sponsored Content
Full Discussion: uniq command
Top Forums UNIX for Advanced & Expert Users uniq command Post 302546204 by Corona688 on Wednesday 10th of August 2011 04:10:55 PM
Old 08-10-2011
What is your input like? Is order important? uniq requires lines in sorted order.

If your input data is small enough to be held in main memory, this may do:

Code:
awk '{ if(!arr[$0]) { arr[$0]=1; print; } }' < data

But if your input data is larger than a few handfuls of megs, this is inefficient, you may have to sort it with
Code:
sort -u

which will sort it and remove duplicates.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

uniq command???

HI, Please tell the usage of uniq command in UNIX with example Thanks (2 Replies)
Discussion started by: skyineyes
2 Replies

2. UNIX for Dummies Questions & Answers

Difference between plain "uniq" and "uniq -u"

Dear all, It's not entirely clear to me from manpage the difference between them. Why we still need "-u" flag? - monkfan (3 Replies)
Discussion started by: monkfan
3 Replies

3. Shell Programming and Scripting

Help with uniq command

I call.... cat 1.txt | uniq -c Sample of whats in 1.txt vmstat cd exit w cd cd cd newgrp xinit f cd cd cd rlogin (2 Replies)
Discussion started by: Bandit390
2 Replies

4. UNIX for Dummies Questions & Answers

Uniq command behaving odd

My file has k s j v l k a s f l k s a d f j l a s (3 Replies)
Discussion started by: phoenix_nebula
3 Replies

5. UNIX for Dummies Questions & Answers

Several file comparison not uniq or comm command

When comparing several files is there a way to find values unique to each file? File1 a b c d File2 a b t File 3 a (3 Replies)
Discussion started by: dr_sabz
3 Replies

6. Shell Programming and Scripting

Uniq Command?

Hello, I have a file with a list of car makes and specific information for each make. An example is: @Audi:Warranty @Audi:Pricing @Audi:Colors @Acura:Warranty @Acura:Pricing @Acura:Colors and so on through a bunch of makes. I need to make a list in a selection box of just one name of... (12 Replies)
Discussion started by: macbb1117
12 Replies

7. UNIX for Dummies Questions & Answers

Re: How To Use UNIQ UNIX Command On single Column

Hi , Can You Please let Know How use unix uniq command on a single column for deleting records from file with Below Structure.Pipe Delimter File . Source Name | Account_Id A | 101 B... (2 Replies)
Discussion started by: anudeepkumar123
2 Replies

8. Shell Programming and Scripting

Addition as part of uniq command

Dear Friends, I want to know if it is possible to perform addition as part of uniq command. Input: 213.64.56.208 1 213.64.56.208 2 213.64.56.208 3 213.46.27.204 10 213.46.27.204 20 213.46.27.204 30 The above input should be converted as follows 213.64.56.208 6 213.46.27.204 60 ... (4 Replies)
Discussion started by: tamil.pamaran
4 Replies

9. Shell Programming and Scripting

cat and uniq command

Hello i have tried to retrieve a distinct list of data from a file with a command combination but the uniq command line is not filtering the fields repeated. Can someone help me ? here is the file input 3837734|LAUNCH TEST01 3837735|LAUNCH TEST01 3837736|LAUNCH TEST01 3837737|LAUNCH... (4 Replies)
Discussion started by: ade05fr
4 Replies

10. Shell Programming and Scripting

Trying to find the distinct lines using uniq command

Platform :Oracle Linux 6.4 Shell : bash The below file has 7 lines , some of them are duplicates. There are only 3 distinct lines. But why is the uniq command still showing 7 ? I just want the distinct lines to be returned. $ cat test.txt SELECT FC.COORD_SET_ID FROM OM_ORDER_FLOW F, -... (2 Replies)
Discussion started by: kraljic
2 Replies
UNIQ(1) 						    BSD General Commands Manual 						   UNIQ(1)

NAME
uniq -- report or filter out repeated lines in a file SYNOPSIS
uniq [-c | -d | -u] [-i] [-f num] [-s chars] [input_file [output_file]] DESCRIPTION
The uniq utility reads the specified input_file comparing adjacent lines, and writes a copy of each unique input line to the output_file. If input_file is a single dash ('-') or absent, the standard input is read. If output_file is absent, standard output is used for output. The second and succeeding copies of identical adjacent input lines are not written. Repeated lines in the input will not be detected if they are not adjacent, so it may be necessary to sort the files first. The following options are available: -c Precede each output line with the count of the number of times the line occurred in the input, followed by a single space. -d Only output lines that are repeated in the input. -f num Ignore the first num fields in each input line when doing comparisons. A field is a string of non-blank characters separated from adjacent fields by blanks. Field numbers are one based, i.e. the first field is field one. -s chars Ignore the first chars characters in each input line when doing comparisons. If specified in conjunction with the -f option, the first chars characters after the first num fields will be ignored. Character numbers are one based, i.e. the first character is character one. -u Only output lines that are not repeated in the input. -i Case insensitive comparison of lines. DIAGNOSTICS
The uniq utility exits 0 on success, and >0 if an error occurs. COMPATIBILITY
The historic +number and -number options have been deprecated but are still supported in this implementation. SEE ALSO
sort(1) STANDARDS
The uniq utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compatible. HISTORY
A uniq command appeared in Version 3 AT&T UNIX. BSD
June 6, 1993 BSD
All times are GMT -4. The time now is 09:19 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy