UNIQ(1) General Commands Manual UNIQ(1)NAME
uniq - delete consecutive identical lines in a file
SYNOPSIS
uniq [-cdu] [-n] [+n] [input [output]]
OPTIONS -c Give count of identical lines in the input
-d Only duplicate lines are written to output
-u Only unique lines are written to output
-n Skip the first n columns when matching
+n Skip the first n fields when matching
EXAMPLES
uniq +2 file # Ignore first 2 fields when comparing
uniq -d inf outf # Write duplicate lines to outf
DESCRIPTION
Uniq examines a file for consecutive lines that are identical. All but duplicate entries are deleted, and the file is written to output.
The +n option skips the first n fields, where a field is defined as a run of characters separated by white space. The -n option skips the
first n spaces. Fields are skipped first.
SEE ALSO sort(1).
UNIQ(1)
Check Out this Related Man Page
UNIQ(1) FSF UNIQ(1)NAME
uniq - remove duplicate lines from a sorted file
SYNOPSIS
uniq [OPTION]... [INPUT [OUTPUT]]
DESCRIPTION
Discard all but one of successive identical lines from INPUT (or standard input), writing to OUTPUT (or standard output).
Mandatory arguments to long options are mandatory for short options too.
-c, --count
prefix lines by the number of occurrences
-d, --repeated
only print duplicate lines
-D, --all-repeated[=delimit-method] print all duplicate lines
delimit-method={none(default),prepend,separate} Delimiting is done with blank lines.
-f, --skip-fields=N
avoid comparing the first N fields
-i, --ignore-case
ignore differences in case when comparing
-s, --skip-chars=N
avoid comparing the first N characters
-u, --unique
only print unique lines
-w, --check-chars=N
compare no more than N characters in lines
--help display this help and exit
--version
output version information and exit
A field is a run of whitespace, then non-whitespace characters. Fields are skipped before chars.
AUTHOR
Written by Richard Stallman and David MacKenzie.
REPORTING BUGS
Report bugs to <bug-coreutils@gnu.org>.
COPYRIGHT
Copyright (C) 2002 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICU-
LAR PURPOSE.
SEE ALSO
The full documentation for uniq is maintained as a Texinfo manual. If the info and uniq programs are properly installed at your site, the
command
info uniq
should give you access to the complete manual.
uniq (coreutils) 4.5.3 February 2003 UNIQ(1)
Hi,
May I know, if a pipe separated File is large, what is the best method to calculate the unique row count of 3rd column and get a list of unique value of the 3rdcolum?
Thanks in advance! (20 Replies)
Dear All,
I want to get help for below case.
I have a file like this.
saman 1
gihan 2
saman 4
ravi 1
ravi 2
so i want to get the result,
saman 5
gihan 2
ravi 3 like this.
Pls help me. (17 Replies)
Hi All,
I am trying to extract data from a large text file , I want to extract lines which contains a five digit number followed by a hyphen , like
12345- , i tried with egrep ,eg : egrep "+" text.txt
but which returns all the lines which contains any number of digits followed by hyhen ,... (19 Replies)
Hi,
I have a table in Db2 with data say
id_1 phase1
id_1 phase2
id_1 phase3
id_2 phase1
id_2 phase2
I need to concatenate the values like
id_1 phase1,phase2,phase3
id_2 phase1,phase2
I tried recursive query but in vain as the length of string to be concatenated in quite long. ... (17 Replies)
hi,
is it possible to find the number of occurences of a pattern between two paranthesis.
for e.g
i have a file as below.
>>{
>>hi
>>GoodMorning
>>how are you?
>>}
>>is it good,
>>tell me yes, if it is good
In the above file, its clear the occurence of word "Good"... (17 Replies)
Hi All,
Is there a way to get NON unique lines from 2 or more concatenated files?
Basically I have several files which are very similar with the exception of few lines and I want to find out which lines are different in each file. Very simple example is
file1 contains:
1
2
3
4
5file2... (122 Replies)
Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example:
input:
<string-array name="threeItems">
<item>item1</item>
<item>item2</item>
<item>item3</item>
</string-array>
<string-array name="twoItems">
<item>item1</item>
<item>item2</item>... (19 Replies)
Hi friends,
I have multiple files. For now, let's say I have two of the following style
cat 1.txt
cat 2.txt
output.txt
Please note that my files are not sorted and in the output file I need another extra column that says the file from which it is coming. I have more than 100... (19 Replies)
Hi,
I have the following command in place
nawk -F, '!a++' file > file.uniq
It has been working perfectly as per requirements, by removing duplicates by taking into consideration only first 3 fields. Recently it has started giving below error:
bash-3.2$ nawk -F, '!a++'... (17 Replies)
Hi i want to fetch 100k record from a file which is looking like as below.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
... (17 Replies)
Hello Everyone!
I have updated the first post so that my intentions are easier to understand, and also attached sample files (post #18).
I have over 500 text files in a directory. Over 1 GB of data. The data in those files is organised in lines:
My intention is to return one line per... (23 Replies)
I am trying to add a unique identifier to two file extensions .bam and .vcf in a directory located at /home/cmccabe/Desktop/index/R_2016_09_21_14_01_15_user_S5-00580-9-Medexome.
The identifier is in $2 of the input file. What the code below is attempting to do is strip off the last portion... (21 Replies)