UNIX for Dummies Questions & Answers

I have a large number (50,000) of pretty large compressed files and I need only certain lines of data from them (each relevant line contains a certain key word). Each file contains 300 such lines. The individual file names are indexed by file number (file_name.1, file_name.2, ... , file_name.50000).

So, I need to uncompress, pull out relevant lines and write to combined_file_name and compress again. I wrote the following bash script to do this:

Code:

i=1
num=50000
while [ $i -le $num ]
  do
  bunzip2 file_name.$i.bz2
  grep key_word file_name.$i >> combined_file_name
  bzip2 file_name.$i
  i=$[$i+1]
done

What I am wondering is whether there is a better/faster way to accomplish this.

Any advice would be much appreciated.

Thank you

---------- Post updated at 12:18 PM ---------- Previous update was at 11:50 AM ----------

While reading another post on here, I just discovered the bzcat command.

I am guessing something like this would be faster:

Code:

i=1
num=50000
while [ $i -le $num ]
  do
  bzcat file_name.$i.bz2 | grep key_word >> combined_file_name
  i=$[$i+1]
done

Is there something even better?

Thanks again in advance.

Thank you for the advice. I guess I don't have bzgrep installed. I tried using zgrep, but that is actually slower than bzcat piped to grep.

The latter is definitely much faster than my original loop, but still too slow.

I am quite new to all this, so sorry if I am sounding rather amateurish (I am indeed that).

In each of my files, the relevant lines are always the same ones - as a means of an example, say I want to extract lines 5, 17, 38, ... from each of the files. So supposing I can identify which the real line numbers are, would there be a better way to extract them than to use grep. It seems not very efficient to have grep search every file if in effect it is known exactly which lines are needed.

Thank you again and sorry if there is an obvious answer.