Sponsored Content
Top Forums Shell Programming and Scripting Filter uniq field values (non-substring) Post 302900619 by alister on Wednesday 7th of May 2014 11:01:49 PM
Old 05-08-2014
Since the strings tested aren't regular expressions, using the regular expression operator is, at best, unnecessarily expensive. At worst, if the strings are allowed to contain regular expression metacharacters, it can lead to an erroneous result.

I suggest using index() instead. For non-trivial data sets, it will also speed things up dramatically.

Testing a near-worst case scenario. The file contains 1501 lines and only the last line contains a string which is a substring of another. Note that while gawk is used, testing with mawk and nawk showed similar improvements:
Code:
$ yes | awk 'NR>1500 {exit} {print NR, NR+1000} END {print NR, 25}' > 1501_1.txt
$ tail -n5 1501_1.txt 
1497 2497
1498 2498
1499 2499
1500 2500
1501 25
$ time gawk '{for(i in a) {if (i~$2) next;if ($2 ~ i) delete a[i]};a[$2]=$0}END {for (i in a) print a[i]}' 1501_1.txt | tail -n5
638 1638
269 1269
228 1228
639 1639
229 1229

real	0m10.462s
user	0m10.149s
sys	0m0.276s
$ time gawk '{for(i in a) {if (index(i,$2)) next;if (index($2, i)) delete a[i]};a[$2]=$0}END {for (i in a) print a[i]}' 1501_1.txt | tail -n5
638 1638
269 1269
228 1228
639 1639
229 1229

real	0m0.895s
user	0m0.892s
sys	0m0.004s

Regards,
Alister
These 4 Users Gave Thanks to alister For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Uniq using only the first field

Hi all, I have a file that contains a list of codes (shown below). I want to 'uniq' the file using only the first field. Anyone know an easy way of doing it? Cheers, Dave ##### Input File ##### 1xr1 1xws 1yxt 1yxu 1yxv 1yxx 2o3p 2o63 2o64 2o65 1xr1 1xws 1yxt 1yxv 1yxx 2o3p 2o63 2o64... (8 Replies)
Discussion started by: Digby
8 Replies

2. UNIX for Dummies Questions & Answers

How to uniq third field in a file

Hi ; I have a question regarding the uniq command in unix How do I uniq 3rd field in a file ? original file : zoom coord 39 18652 39 18652 zoom coord 39 18653 39 18653 zoom coord 39 18818 39 18818 zoom coord 39 18840 39 18840 zoom coord 41 15096 41 15096 zoom... (1 Reply)
Discussion started by: babycakes
1 Replies

3. Shell Programming and Scripting

How to use uniq on a certain field?

How can I use uniq on a certain field or what else could I use? If I want to use uniq on the second field and the output would remove one of the lines with a 5. bob 5 hand jane 3 leg jon 4 head chris 5 lungs (1 Reply)
Discussion started by: Bandit390
1 Replies

4. Shell Programming and Scripting

filter the uniq record problem

Anyone can help for filter the uniq record for below example? Thank you very much Input file 20090503011111|test|abc 20090503011112|tet1|abc|def 20090503011112|test1|bcd|def 20090503011131|abc|abc 20090503011131|bbc|bcd 20090503011152|bcd|abc 20090503011151|abc|abc... (8 Replies)
Discussion started by: bleach8578
8 Replies

5. Shell Programming and Scripting

Uniq based on first field

Hi New to unix. I want to display only the unrepeated lines from a file using first field. Ex: 1234 uname1 status1 1235 uname2 status2 1234 uname3 status3 1236 uname5 status5 I used sort filename | uniq -u output: 1234 uname1 status1 1235 uname2 status2 1234 uname3 status3 1236... (10 Replies)
Discussion started by: venummca
10 Replies

6. Shell Programming and Scripting

Sort field and uniq

I have a flatfile A.txt 2012/12/04 14:06:07 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 17:07:22 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 17:13:27 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 14:07:39 |rain|Boards 1|tampa|merced|merced11 How do i sort and get... (3 Replies)
Discussion started by: sabercats
3 Replies

7. Shell Programming and Scripting

Printing uniq first field with the the highest second field

Hi All, I am searching for a script which will produce an output file with the uniq first field with the second field having highest value among all the duplicates.. The output file will produce only the uniqs which are duplicate 3 times.. Input file X 9 B 5 A 1 Z 9 T 4 C 9 A 4... (13 Replies)
Discussion started by: ailnilanjan
13 Replies

8. Shell Programming and Scripting

Grok filter to extract substring from path and add to host field in logstash

Hii, I am reading data from files by defining path as *.log etc, Files names are like app1a_test2_heep.log , cdc2a_test3_heep.log etc How to configure logstash so that the part of string that is string before underscore (app1a, cdc2a..) should be grepped and added to host field and... (7 Replies)
Discussion started by: Ravi Kishore
7 Replies

9. Shell Programming and Scripting

HELP - uniq values per column

Hi All, I am trying to output uniq values per column. see file below. can you please assist? Thank you in advance. cat names joe allen ibm joe smith ibm joe allen google joe smith google rachel allen google desired output is: joe allen google rachel smith ibm (5 Replies)
Discussion started by: Apollo
5 Replies

10. Shell Programming and Scripting

awk to update field using matching value in file1 and substring in field in file2

In the awk below I am trying to set/update the value of $14 in file2 in bold, using the matching NM_ in $12 or $9 in file2 with the NM_ in $2 of file1. The lengths of $9 and $12 can be variable but what is consistent is the start pattern will always be NM_ and the end pattern is always ;... (2 Replies)
Discussion started by: cmccabe
2 Replies
AGREP(1)						    BSD General Commands Manual 						  AGREP(1)

NAME
agrep -- print lines approximately matching a pattern SYNOPSIS
agrep [options] pattern [files] DESCRIPTION
Searches for approximate matches of pattern in each FILE or standard input. OPTIONS
Regexp selection and interpretation -e pattern, --regexp=pattern Use PATTERN as a regular expression; useful to protect patterns beginning with '-'. -i, --ignore-case Ignore case distinctions (as defined by the current locale) in pattern and input files. -k, --literal Treat pattern as a literal string, that is, a fixed string with no special characters. -w, --word-regexp Force pattern to match only whole words. A ``whole word'' is a substring which either starts at the beginning or the record or is preceded by a non-word constituent character. Similarly, the substring must either end at the end of the record or be fol- lowed by a non-word constituent character. Word-constituent characters are alphanumerics (as defined by the current locale) and the underscore character. Note that the non-word constituent characters must surround the match; they cannot be counted as errors. Approximate matching settings -D num, --delete-cost=num Set cost of missing characters to num. -I num, --insert-cost=num Set cost of extra characters to num. -S num, --substitue-cost=num Set cost of incorrect characters to num. Note that a deletion (a missing character) and an insertion (an extra character) together constitute a substituted character, but the cost will be the that of a deletion and an insertion added together. Thus, if the const of a substitution is set to be larger than the sum of the costs of deletion and insertion, direct substitutions will never be done. -E -num, --max-errors=num Select records that have at most num errors. -# Select records that have at most # errors (# is a digit between 0 and 9). Miscellaneous -d -pattern, --delimiter=pattern Set the record delimiter regular expression to pattern. The text between two delimiters, before the first delimiter, and after the last delimiter is considered to be a record. The default record delimiter is the regexp `` '', so by default a record is a line. pattern can be any regular expression that does not match the empty string. For example, using -d file ... defines mail messages as records in a Mailbox format file. -v, --invert-match Select non-matching records instead of matching records. -V, --version Print version information and exit. -y, --nothing Does nothing. This options exists only for compatibility with the non-free agrep program. --help Display a brief help message and exit. Output control -B, --best-match Only output the best matching records, that is, the records with the lowest cost. This is currently implemented by making two passes over the input files and cannot be used when reading from standard input. --color, --colour Highlight the matching strings in the output with a color marker. The color string is taken from the GREP_COLOR environment variable. The default color is red. -c, --count Only print a count of matching records per each input file, suppressing normal output. -h, --no-filename Suppress the prefixing filename on output when multiple files are searched. -H, --with-filename Prefix each output record with the name of the input file where the record was read from. -l, --files-with-matches Only print the name of each input file which contains at least one match, suppressing normal output. The scanning for each file will stop on the first match. -n, --record-number Prefix each output record with its sequence number in the input file. The number of the first record is 1. -q, --quiet, --silent Do not write anything to standard output. Exit immediately with zero exit status if a match is found. -s, --show-cost Print match cost with output. --show-position Prefix each output record with the start and end offset of the first match within the record. The offset of the first character of the record is 0. The end position is given as the offset of the first character after the match. -M, --delimiter-after By default, the record delimiter is the newline character and is output after the matching record. If -d is used, the record delimiter will be output before the matching record. This option causes the delimiter to be output after the matching record. With no file, or when file is ``-'', agrep reads standard input. If less than two files are given -h is assumed, otherwise -H is the default. EXAMPLES
agrep -2 optimize foo.txt outputs all lines in file foo.txt that match ``optimize'' within two errors. E.g. lines which contain ``optimise'', ``optmise'', and ``opitmize'' all match. DIAGNOSTICS
Exit status is 0 if a match is found, 1 for no match, and 2 if there were errors. If -E or -# is not specified, only exact matches are selected. pattern is a POSIX extended regular expression (ERE) with the TRE extensions. REPORTING BUGS
Report bugs to the TRE mailing list <tre-general@lists.laurikari.net>. COPYRIGHT
Copyright (C) 2002-2004 Ville Laurikari. This is free software, and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute this software under certain conditions; see the source for the full license text. BSD
November 21, 2004 BSD
All times are GMT -4. The time now is 01:04 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy