02-16-2012
Quote:
Originally Posted by
dinesh1178
I tried copying the lines mentioned here in a text file and used sort. It worked.
You can use uniq to remove duplicate lines.
This may lead to erroneous behaviour : for example if a record contains a single line that already exists in a multi-line record, the singled line record will not be kept eventhough it is uniq from a record point of view.
Sorting and filtering with uniq would not fit with the requirements (as far as i understand them)
9 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi
Just wondering whether or not I can remove duplicated lines without sort
For example, I use the command who, which shows users who are logging on. In some cases, it shows duplicated lines of users who are logging on more than one terminal.
Normally, I would do
who | cut -d" " -f1 |... (6 Replies)
Discussion started by: lalelle
6 Replies
2. Shell Programming and Scripting
Hi Guys...
Please Could you help me with the following ?
aaaa bbbb cccc sdsd
aaaa bbbb cccc qwer
as you can see, the 2 lines are matched in three fields...
how can I delete this pupicate ? I mean to delete the second one if 3 fields were duplicated ?
Thanks (14 Replies)
Discussion started by: yahyaaa
14 Replies
3. UNIX for Dummies Questions & Answers
Hello all,
I've got a strange behaviour of sort and uniq commands: they do not recognise apparently duplicated lines in a file (already sorted). The lines are identical by eye, but they must differ in smth, because when they are put in two files, those have slightly different size.
What can make... (8 Replies)
Discussion started by: roussine
8 Replies
4. Shell Programming and Scripting
We have an input file as follows:
2010-09-15-12.41.15
2010-09-15-12.41.15
2010-09-15-12.41.24
2010-09-15-12.41.24
2010-09-15-12.41.24
2010-09-15-12.41.24
2010-09-15-12.41.25
2010-09-15-12.41.26
2010-09-15-12.41.26
2010-09-15-12.41.26
2010-09-15-12.41.26
2010-09-15-12.41.26... (3 Replies)
Discussion started by: ux4me
3 Replies
5. Shell Programming and Scripting
Dear All,
Suppose I have a file:
1 1 1 1
2 2 2 2
3 3 3 3I want to insert new line under each old line so that the file would become:
1 1 1 1
1 1 1 1
2 2 2 2
2 2 2 2
3 3 3 3
3 3 3 3How can this be accomplished using awk (or sed)? (5 Replies)
Discussion started by: littlewenwen
5 Replies
6. Shell Programming and Scripting
Hi, if i have a file like this:
Query=1
a
a
b
c
c
c
d
Query=2
b
b
b
c
c
e
.
.
. (7 Replies)
Discussion started by: the_simpsons
7 Replies
7. Shell Programming and Scripting
Hello,
I am trying to eliminate rows where the first field is duplicated, leaving the row where the last field is "NET".
Data file:
345234|22.34|LST
546543|55.33|LST
793929|98.23|LST
793929|64.69|NET
149593|49.22|LST
Desired output:
345234|22.34|LST
546543|55.33|LST... (2 Replies)
Discussion started by: palex
2 Replies
8. UNIX for Dummies Questions & Answers
Hi Everybody! First post! Totally noobie.
I'm using the terminal to read a poorly formatted book.
The text file contains, in the middle of paragraphs, hyphenation to split words that are supposed to be on multiple pages. It looks ve -- ry much like this.
I was hoping to use grep -v " -- "... (5 Replies)
Discussion started by: AxeHandle
5 Replies
9. Shell Programming and Scripting
Dear all,
I always appreciate your help.
I would like to delete lines containing duplicated strings in the second column.
test.txt
658 invert_d2e_q_reg_0_/Qalu_ecl_zlow_e 0.825692
659 invert_d2e_q_reg_0_/Qalu_byp_rd_data_e 0.825692
660 invert_d2e_q_reg_0_/Qalu_byp_rd_data_e 0.825692... (1 Reply)
Discussion started by: jypark22
1 Replies
uniq(1) General Commands Manual uniq(1)
Name
uniq - report repeated lines in a file
Syntax
uniq [-udc[+n][-n]] [input[output]]
Description
The command reads the input file comparing adjacent lines. In the normal case, the second and succeeding copies of repeated lines are
removed; the remainder is written on the output file. Note that repeated lines must be adjacent in order to be found. For further infor-
mation, see
Options
The n arguments specify skipping an initial portion of each line in the comparison:
-n Skips specified number of fields. A field is defined as a string of non-space, non-tab characters separated by tabs and spaces from its
neighbors.
+n Skips specified number of characters in addition to fields. Fields are skipped before characters.
-c Displays number of repetitions, if any, for each line.
-d Displays only lines that were repeated.
-u Displays only unique (nonrepeated) lines.
If the -u flag is used, just the lines that are not repeated in the original file are output. The -d option specifies that one copy of
just the repeated lines is to be written. The normal mode output is the union of the -u and -d mode outputs.
The -c option supersedes -u and -d and generates an output report in default style but with each line preceded by a count of the number of
times it occurred.
See Also
comm(1), sort(1)
uniq(1)