in order to process all lines in which the first 16 characters are duplicated.
Now I want to also run that script on a BSD based system where the included version of uniq does not support the -w (--check-chars) option. To get around this I have written an awk script to instead use if GNU uniq is not available. It seems to work with both GNU and BSD versions of awk, but it is pretty ugly.
I am wondering if this can be simplified, or whether there is another, better solution.
Thanks.
Last edited by mij; 10-23-2012 at 10:56 AM..
Reason: Broke up line as stretching page.
Hi,
I have this scenario; where there are two classes:- apple and orange.
1,2,3,4,5,6,apple
1,1,0,4,2,3,apple
1,3,3,3,3,4,apple
1,1,1,1,1,1,orange
1,2,3,1,1,1,orange
Basically for apple, i have 3 entries in the file, and for orange, I have 2 entries. Im trying to edit the file and find... (5 Replies)
Hi, my dilemna is this:
example i got a file of fruit.txt which contains:
Apple 6
Apple_new 7
old_orange 9
orange 10
Is there any way for me to have an output of
Apple 13
Orange 19
using shell script: (6 Replies)
Hello,
I have a log file with the following input:
X , ID , Date, Time, Y
01,01368,2010-12-02,09:07:00,Pass
01,01368,2010-12-02,10:54:00,Pass
01,01368,2010-12-02,13:07:04,Pass
01,01368,2010-12-02,18:54:01,Pass
01,01368,2010-12-03,09:02:00,Pass
01,01368,2010-12-03,13:53:00,Pass... (12 Replies)
Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example:
input:
<string-array name="threeItems">
<item>item1</item>
<item>item2</item>
<item>item3</item>
</string-array>
<string-array name="twoItems">
<item>item1</item>
<item>item2</item>... (19 Replies)
Hi
My file have 7 column, FIle is pipe delimed
Col1|Col2|col3|Col4|col5|Col6|Col7
I want to find out uniq record count on col3, col4 and col2 ( same order) how can I achieve it.
ex
1|3|A|V|C|1|1
1|3|A|V|C|1|1
1|4|A|V|C|1|1
Output should be
FREQ|A|V|3|2
FREQ|A|V|4|1
Here... (5 Replies)
I met a challenge to filter ~70 millions of sequence rows and I want using awk with conditions:
1) longest string of each pattern in column 2, ignore any sub-string, as the index;
2) all the unique patterns after 1);
3) print the whole row;
input:
1 ABCDEFGHI longest_sequence1
2 ABCDEFGH... (12 Replies)
Hi again,
I have files with the following contents
datetime,ip1,port1,ip2,port2,number
How would I find out how many times ip1 field shows up a particular file? Then how would I find out how many time ip1 and port 2 shows up?
Please mind the file may contain 100k lines. (8 Replies)
Hi all
I was wondering if you may help me in resolving an issue.
In particular I have a file like this:
the ... represent different string and what I wrote Cur or Ent are the constant.
Well, what I would like to obtain is a file in which are reported only the ID in which the second column... (6 Replies)
Hi Help,
I have a file which looks like
1 20 30 40 50 60 6
2 20 30 40 50 60 8
7 20 30 40 50 60 7
4 30 40 50 60 70 8
5 30 40 50 60 70 9
2 30 40 50 60 70 8
I want the o/p as
1 20 30 40 50 60 6
4 30 40 50 60 70 8
Is there a way I can use uniq command or awk to do this?
... (11 Replies)
Discussion started by: Indra2011
11 Replies
LEARN ABOUT CENTOS
uniq
UNIQ(1) User Commands UNIQ(1)NAME
uniq - report or omit repeated lines
SYNOPSIS
uniq [OPTION]... [INPUT [OUTPUT]]
DESCRIPTION
Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or standard output).
With no options, matching lines are merged to the first occurrence.
Mandatory arguments to long options are mandatory for short options too.
-c, --count
prefix lines by the number of occurrences
-d, --repeated
only print duplicate lines, one for each group
-D, --all-repeated[=METHOD]
print all duplicate lines groups can be delimited with an empty line METHOD={none(default),prepend,separate}
-f, --skip-fields=N
avoid comparing the first N fields
--group[=METHOD]
show all items, separating groups with an empty line METHOD={separate(default),prepend,append,both}
-i, --ignore-case
ignore differences in case when comparing
-s, --skip-chars=N
avoid comparing the first N characters
-u, --unique
only print unique lines
-z, --zero-terminated
end lines with 0 byte, not newline
-w, --check-chars=N
compare no more than N characters in lines
--help display this help and exit
--version
output version information and exit
A field is a run of blanks (usually spaces and/or TABs), then non-blank characters. Fields are skipped before chars.
Note: 'uniq' does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use 'sort -u' without
'uniq'. Also, comparisons honor the rules specified by 'LC_COLLATE'.
GNU coreutils online help: <http://www.gnu.org/software/coreutils/> Report uniq translation bugs to <http://translationproject.org/team/>
AUTHOR
Written by Richard M. Stallman and David MacKenzie.
COPYRIGHT
Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
SEE ALSO comm(1), join(1), sort(1)
The full documentation for uniq is maintained as a Texinfo manual. If the info and uniq programs are properly installed at your site, the
command
info coreutils 'uniq invocation'
should give you access to the complete manual.
GNU coreutils 8.22 June 2014 UNIQ(1)