05-19-2011
Print duplicate lines
I have a file where some of the lines are duplicates.
How do I use bash to print all the lines that have duplicates?
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have file where it contains 2 columns. In two columns the first column is repeated more than once. I wanted to take the unique record in first column
and the corresponding second column value .
The below is the example of the file:
8244100320012955|000b063471a4... (4 Replies)
Discussion started by: ravi_rn
4 Replies
2. UNIX for Dummies Questions & Answers
OK, I have read several things on how to do this, but can't make it work. I am writing this to a vi file then calling it as an awk script.
So I need to search a file for duplicate lines, delete duplicate lines, then write the result to another file, say /home/accountant/files/docs/nodup
... (2 Replies)
Discussion started by: bfurlong
2 Replies
3. Shell Programming and Scripting
Hi all,
This should be very easy but I can't figure it out...
I have a file that looks like this:
@SRR057408.1 FW8Y5CK02R652T length=34
AGCAGTGGTATCAACGCAGAGTAAGCAGTGGTAT
+SRR057408.1 FW8Y5CK02R652T length=34
FIIHFF6666?=:88@@@BBD:::?@ABBAAA>8
@SRR057408.2 FW8Y5CK02TBMHV length=52... (1 Reply)
Discussion started by: kmkocot
1 Replies
4. Shell Programming and Scripting
abc.dat
tty cpu
tin tout us sy wt id
0 0 7 3 19 71
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 133.2 0.0 682.9 0.0 1.0 0.0 7.2 0 79 c1t0d0
0.2 180.4 0.1 5471.2 3.0 2.8 16.4 15.6 15 52 aaaaaa1-xx
I want to skip first 5 line... (4 Replies)
Discussion started by: kchinnam
4 Replies
5. UNIX for Dummies Questions & Answers
input
chr1 10 100
chr1 10 100
chr1 20 200
output
chr1 10 100 2
chr1 20 200 1 (1 Reply)
Discussion started by: repinementer
1 Replies
6. Shell Programming and Scripting
input
output
a1 100 200 XYZ_X
a1 98 188 ABC (2 Replies)
Discussion started by: quincyjones
2 Replies
7. Shell Programming and Scripting
The input file:
>cat module1
200611051053 95
200523457498 35
200617890187 57
200726098123 66
200645676712 71
200744556590 68
>cat module2
200645676712 ... (10 Replies)
Discussion started by: winter9
10 Replies
8. UNIX for Advanced & Expert Users
Hi All,
I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space.
I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies
9. Shell Programming and Scripting
hi,
my input
a b c d e f g
k i m g i k l
b i n g o h m
a b c d e
b i n g o
my output
a b c d e f g
k i m g i k l
b i n g o h m
what I would like to do is compare the first five columns of each row to the entire records in the file and if there is a match, eliminate that... (1 Reply)
Discussion started by: jacobs.smith
1 Replies
10. UNIX for Beginners Questions & Answers
I have a directory of files, I can show the number of lines in each file and order them from lowest to highest with:
wc -l *|sort
15263 Image.txt
16401 reference.txt
40459 richtexteditor.txt
How can I also print the number of unique lines in each file?
15263 1401 Image.txt
16401... (15 Replies)
Discussion started by: spacegoose
15 Replies
LEARN ABOUT LINUX
msguniq
MSGUNIQ(1) GNU MSGUNIQ(1)
NAME
msguniq - unify duplicate translations in message catalog
SYNOPSIS
msguniq [OPTION] [INPUTFILE]
DESCRIPTION
Unifies duplicate translations in a translation catalog. Finds duplicate translations of the same message ID. Such duplicates are invalid
input for other programs like msgfmt, msgmerge or msgcat. By default, duplicates are merged together. When using the --repeated option,
only duplicates are output, and all other messages are discarded. Comments and extracted comments will be cumulated, except that if
--use-first is specified, they will be taken from the first translation. File positions will be cumulated. When using the --unique
option, duplicates are discarded.
Mandatory arguments to long options are mandatory for short options too.
Input file location:
INPUTFILE
input PO file
-D, --directory=DIRECTORY
add DIRECTORY to list for input files search
If no input file is given or if it is -, standard input is read.
Output file location:
-o, --output-file=FILE
write output to specified file
The results are written to standard output if no output file is specified or if it is -.
Message selection:
-d, --repeated
print only duplicates
-u, --unique
print only unique messages, discard duplicates
Input file syntax:
-P, --properties-input
input file is in Java .properties syntax
--stringtable-input
input file is in NeXTstep/GNUstep .strings syntax
Output details:
-t, --to-code=NAME
encoding for output
--use-first
use first available translation for each message, don't merge several translations
--color
use colors and other text attributes always
--color=WHEN
use colors and other text attributes if WHEN. WHEN may be 'always', 'never', 'auto', or 'html'.
--style=STYLEFILE
specify CSS style rule file for --color
-e, --no-escape
do not use C escapes in output (default)
-E, --escape
use C escapes in output, no extended chars
--force-po
write PO file even if empty
-i, --indent
write the .po file using indented style
--no-location
do not write '#: filename:line' lines
-n, --add-location
generate '#: filename:line' lines (default)
--strict
write out strict Uniforum conforming .po file
-p, --properties-output
write out a Java .properties file
--stringtable-output
write out a NeXTstep/GNUstep .strings file
-w, --width=NUMBER
set output page width
--no-wrap
do not break long message lines, longer than the output page width, into several lines
-s, --sort-output
generate sorted output
-F, --sort-by-file
sort output by file location
Informative output:
-h, --help
display this help and exit
-V, --version
output version information and exit
AUTHOR
Written by Bruno Haible.
REPORTING BUGS
Report bugs to <bug-gnu-gettext@gnu.org>.
COPYRIGHT
Copyright (C) 2001-2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
SEE ALSO
The full documentation for msguniq is maintained as a Texinfo manual. If the info and msguniq programs are properly installed at your
site, the command
info msguniq
should give you access to the complete manual.
GNU gettext-tools 0.18.1 June 2010 MSGUNIQ(1)