Shell Programming and Scripting

We have an egrep search in a while loop.

egrep -w "$key" ${PICKUP_DIR}/new_update >> ${PICKUP_DIR}/update_record_new

${PICKUP_DIR}/new_update is 210 MB file

In each iteration, the egrep on an average takes around 50-60 seconds to search. Ther'es nothing significant in the loop other than egrep. And when we checked the timestamps, egrep is what slowing it down.

Is it possible to improve egrep's performance ? Or do we need to use perl or any other pattern search ?

Could you please help ?

Does the value of "key" and "PICKUP_DIR" change with each iteration ?

Look into the -f flag of grep.

The value of $key changes on each iteration but ${PICKUP_DIR}/new_update doesn't change

So look into the -f flag.

Code:

egrep -f <file containing the different values of $key> ${PICKUP_DIR}/new_update

In addition to the above, can you post an example of this $key? Perhaps using a regex optimizer will help. If readability to external assembly of a $key is done, you could do something like this as well.

Code:

grep -E -w "`regexopt $key`" ...

I have uploaded the $key as a screenshot as I don't have the text version right now..., it's a big string concatenated by "|".

Can you pls. tell me which is better than egrep....
grep.. perl... sed...?
And why should egrep take around 50..60 seconds in an iteration ...?
And will splitting the ${PICKUP_DIR}/new_update file into multiple files and searching each file until a match is found, help in anyway...?

Are the keys separated by a '|' ? Or is the whole thing a key in itself ?

If the keys are separated by '|', then change the file such that each key is on a new line. Then

Code:

egrep -f key.txt ${PICKUP_DIR}/new_update

I dont know if you will have any advantage in splitting up the file.