Shell Programming and Scripting

Hi.

I created a 12 MB file that had the first pattern at the end of the file, and the other not present.

I timed 5 solutions: alister's suggested 2-grep, a similar method but with the more-featured cgrep, rapgrep (a perl code: "require all patterns"), glark (a Ruby code), and an awk script. The 2-grep, 2-cgrep, and awk were the fastest in that order, far faster than the perl and Ruby codes.

That 2 passes through the file would be faster than a single pass surprised me. The grep family appears to be very well-written, as is the gawk processor ... cheers, drl

Did you happen to test my suggestions as well? In brief testing I found them to be even faster, possibly due to the fact that the grep moves on to the next file after the first match and only those files that contain a match for the first pattern are grepped for the second...

S.

Hi, Scrutinizer.

In a way I did.

The "optimized" part of the timed code for the grep segment went like this:

Code:

grep -q -m 1 "$pattern1" "$FILE" && grep -q -m 1 "$pattern2" "$FILE"

I think that probably does much the same thing as your suggestion. I used alister's suggestion as the base.

Note that in the case of the 12 MB file, this probably does not matter because the first string to be matched is specifically placed in the last line forcing the first grep to go all the way through, succeeding, and then the second grep to take place, failing -- a worst-case situation. However, either the -l option or the -m 1 option should work the same -- i.e. bail out at the first match, although I admit that I did not compare one to the other. In a production environment, one or the other should be used to avoid wasted time.

I did something similar for cgrep -- the option differs in syntax but is the same semantically.

I did not use the glark feature to run recursively because I wanted most of the infrastructure to be the same -- find, xargs, etc. The find, xargs construction accounts for another possible worst-case, where one might have many (too many) files that pass the first test.

Thanks for your observations & feedback ... cheers, drl

Hi drl,

Thanks for your observations & feedback in return. I guess probably the observed difference stems from a continual call to the grep program as part of the loop vs. 2 times a single call to grep scanning all required files in one go...

S.

---------- Post updated 14-06-10 at 08:16 ---------- Previous update was 13-06-10 at 23:58 ----------

Grep is very efficient indeed. Also have a look at mawk. In my experience it usually beats gawk on speed...

Hi.

Indeed, mawk is slightly faster than gawk. The mawk variant is occasionally mentioned in comp.lang.awk. It is available in the Debian repositories, so I often have it available, but I rarely install it in other distributions that I have in (testing installs on) virtual machines. For example, it s not present in the standard repositories for openSUSE 11.2 "Emerald".

Both claim to be standard compliant, although the exact phrase for gawk is " ... almost completely POSIX 1003.2 compliant ..."

Here are the versions and timings for one run against a 12 MB file that has "fiber" only at the very end of the file, and "alpha" does not occur at all -- using the techniques discussed earlier:

Code:

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 
GNU bash 3.2.39
cgrep - (local: ~/executable/cgrep May 29 2009 )
glark, version 1.8.0
rapgrep (local) 1.2
GNU grep 2.5.3
GNU Awk 3.1.5
mawk - ( /usr/bin/mawk Apr 5 2008 )

 Results of finding "fiber" and "alpha" contained in files, grep:

real	0m0.088s
user	0m0.036s
sys	0m0.036s

-----
 Results of finding "fiber" and "alpha" contained in files, cgrep:
fiber

real	0m0.118s
user	0m0.052s
sys	0m0.032s

-----
 Results of finding "fiber" and "alpha" contained in files, rapgrep:

real	0m13.930s
user	0m11.309s
sys	0m0.020s

-----
 Results of finding "fiber" and "alpha" contained in files, glark:

real	0m11.703s
user	0m8.865s
sys	0m0.696s

-----
 Results of finding "fiber" and "alpha" contained in files, mawk:

real	0m0.176s
user	0m0.140s
sys	0m0.020s

-----
 Results of finding "fiber" and "alpha" contained in files, awk:

real	0m0.300s
user	0m0.260s
sys	0m0.012s

cheers, drl