11-30-2017
So - lines repeated up to three times should be printed as are - once, twice, or thrice. Then, lines repeated four or more times should be printed once?
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I have following file content (3 fields each line):
23 888 10.0.0.1
dfh 787 10.0.0.2
dssf dgfas 10.0.0.3
dsgas dg 10.0.0.4
df dasa 10.0.0.5
df dag 10.0.0.5
dfd dfdas 10.0.0.5
dfd dfd 10.0.0.6
daf nfd 10.0.0.6
...
as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Discussion started by: fredao
3 Replies
2. Shell Programming and Scripting
Hi, I have a huge file which is about 50GB. There are many lines. The file format likes
21 rs885550 0 9887804 C C T C C C C C C C
21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0
21 rs303304 0 9941889 A A A A A A A A A A
22 rs303304 0 9941890 0 A A A A A A A A A
The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies
3. Shell Programming and Scripting
Hi,
I'm trying to create an XML sitemap of our dynamic ecommerce sites SEO Friendly URLs and am trying to create the initial page listing.
I have a CSV file that looks like the following and need duplicate the lines based on a value which needs calculating.
... (2 Replies)
Discussion started by: jamesfx
2 Replies
4. Shell Programming and Scripting
Hello Folks,
Can some one help me with the removal of duplicate lines from a log file and send it to another log file. It's bit complicated as two lines are same but only difference is the timestamp, but some lines are uniq. Line has been seperated by colon's.
Log file:... (5 Replies)
Discussion started by: sim_je
5 Replies
5. Shell Programming and Scripting
How can i delete some regular repeated lines in a file?
example:
in_file
EDGE 1 2 12
EDGE 2 3 23
EDGE 3 4 34
EDGE 5 6 56
EDGE 6 7 67
EDGE 7 8 78
EDGE 9 10 910
EDGE 10 11 1011
EDGE 11 12 1112
EDGE 13 14 1314
EDGE 14 15 1415
EDGE 15 16 1516
EDGE 17 18 1718
EDGE 18 19 1819
EDGE 19... (8 Replies)
Discussion started by: saeed.soltani
8 Replies
6. Shell Programming and Scripting
Hello All, I have a file which has repeated lines. I want to print the lines which are repeated three times.
Please help. (3 Replies)
Discussion started by: ailnilanjan
3 Replies
7. UNIX for Dummies Questions & Answers
Hi
I need this output. Thanks.
Input:
TAZ
YET
FOO
FOO
VAK
TAZ
BAR
Output:
YET
VAK
BAR (10 Replies)
Discussion started by: tara123
10 Replies
8. Shell Programming and Scripting
Within my text file i have several thousand lines of text with some lines containing duplicate strings/words. I would like to entirely remove those lines which contain the duplicate strings.
Eg;
One and a Two
Unix.com is the Best
This as a Line Line
Example duplicate sentence with the word... (22 Replies)
Discussion started by: martinsmith
22 Replies
9. UNIX for Beginners Questions & Answers
Dears
i want to extract lines only that have first entry repeated 3 times or above , ex data :
-bash-3.00$ cat INTCONT-IS.CSV
M205-00-106_AMDRN:1-0-6-22,12-662-4833,intContact,2016-11-15 02:32:16,50
M205-00-106_AMDRN:1-0-23-17,12-616-0462,intContact,2016-11-15 02:32:23,50... (5 Replies)
Discussion started by: is2_egypt
5 Replies
10. Shell Programming and Scripting
Hi All,
I am storing the result in the variable result_text using the below code.
result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines.
file and time for the interval 03:30 - 03:45
file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies
uniq(1) General Commands Manual uniq(1)
NAME
uniq - report repeated lines in a file
SYNOPSIS
fields] chars] [input_file [output_file]]
DESCRIPTION
reads the input text file input_file, comparing adjacent lines, and copies the result to output_file. If input_file is not specified, the
standard input and standard output are used. If input_file is specified, but output_file is not, results are printed to standard output.
input_file and output_file must not be the same file.
Line-Comparison Options
recognizes the following options when comparing adjacent lines:
Print those lines that are repeated in the original file.
Print copy only of each repeated line in the input file.
Generate an output report in default style
except that each line is preceded by a count of the number of times it occurred. If this option is specified, the and
options are ignored if either or both are also present.
If none of the options or are present, prints the results of the union of the and options, producing a copy of the original input file with
the second and succeeding copies of any repeated lines removed. (Note that repeated lines must be adjacent in order to be found -- see
sort(1)).
Field-Skip Options
Two options are provided for skipping an initial portion of each line when making comparisons:
Ignore the first
fields fields, together with any blanks before each. fields is a positive decimal integer. A field is defined as a
string of non-space, non-tab characters separated by tabs and/or spaces from its neighbors.
Ignore the first
chars characters. chars is a positive decimal integer. Each line in the input is assumed to be terminated with a
new line character for purposes of comparison. Fields are skipped before characters.
EXTERNAL INFLUENCES
Environment Variables
must be equal to the value it had when the input files were sorted.
determines the interpretation of text within files as single- and/or multi-byte characters, and defines a space character when the or
option is used.
determines the language in which messages are displayed.
If or is not specified in the environment or is set to the empty string, the value of is used as a default for each unspecified or empty
variable. If is not specified or is set to the empty string, a default of "C" (see lang(5)) is used instead of If any internationalization
variable contains an invalid setting, behaves as if all internationalization variables are set to "C". See environ(5).
International Code Set Support
Single- and multi-byte character code sets are supported.
RETURN VALUE
Exit values are:
0 Successful completion.
>0 Error condition occurred.
AUTHOR
was developed by OSF and HP.
SEE ALSO
comm(1), sort(1).
STANDARDS CONFORMANCE
uniq(1)