Shell Programming and Scripting

i'm working with files that are huge in size. over 3GB. and i need to do a lot of pattern matching. I need a way to grep for what i want, using a tool that is available across most unix systems.

i initially was gungho about grep, but not all capablities of grep are available on all OSes.

so im focusing on a command like this. how can i make it run faster?

Code:

date ; perl -ne 'print if /core.manager.referral/' $DATAFILE | perl -ne 'print if /ERROR getting Referral/' | wc -l ; date

this code searches for all lines that contain both "core.manager.referral " AND "ERROR getting Referral"

shell = bash
os = linux red hat, sunos
datafile = 4GB

Seems all you need is plain-vanilla grep, not any fancy options.
Grep on what you think will have the fewest matches, to make a much smaller file. Then grep on the other option.

Or, are you trying to determine if perl & if is faster than grep?

It would be good if you can post some small part of your input file

yes i'm trying to see if perl is faster than grep.

and posting the input file is not an option at all. and the strings i picked out to search for necessary. i deliberately picked them.

perl is massively huger than grep, that's a good sign it's not going to be faster. Running it twice, that's certainly not going to be faster. Piping it into wc -l is going to be slower yet.

How about awk? Getting rid of pipes and external utilities is a good way to speed up a program, and awk can easily do everything you want here in one program with no pipes or external utilities (besides awk itself of course).

Code:

awk '/core.manager.referral/ && /ERROR getting Referral/ { N++} END { print N }' filename

thank you. i tried this, and the command is still slow to finish.

im open to any one-liner type of command like this that will be the fastest.

You are working with FOUR GIGABYTES of data. Of course it's slow. What's the file size, divided by your maximum disk speed? How much RAM do you have for cache? What's the bus speed of your RAM?

I don't think you're going to improve much on the awk version. You might try tricks with index() instead of regexes.