Shell Programming and Scripting

Hi,

I have 2 large log files in .gz format

file 1 contains

abcde
12345
23456
.
.
.
.
.
.
.
.
09123


file 2
abcde,1,2,3,4,5,6,7
09123,3,4,5,6,7,7,8
23456,9,6,5,4,3,2,1
....
...
...
...

I am basically looking for a script to open the file1 , read line by line and verify if the string matches in file 2 then redirect matching lines to one more file 3 , the string match should be irrespective of position they occur in file2.

Since i have a space limitation can the script be executed in the compressed format only

It would be better if a PERL search script is provided as it is pretty faster than awk.

Not perl , but give it a try:

Code:

gunzip -c file1.dat.gz|while read pat
do
   gunzip -c file2.dat.gz |grep "${pat}" 
done >file3

If you want to compress de destination file:

Code:

gunzip -c file1.dat.gz|while read pat
do
   gunzip -c file2.dat.gz |grep "${pat}" 
done |gzip -c -9  >file3.dat.gz

Thanks but File 1 has 10 million lines and File 2 has 60 million lines , can it run faster with the above script

Yep , its going to be always slow (because of requirements) , 10 M x 60 M makes 600 M of file2 "full scans" ...

Actually, it can get considerably faster. The above solution uses a shell loop which is very slow. Look here at a very similar thread:
https://www.unix.com/shell-programmin...hing-perl.html

Yes , if you're plenty of memory

True, memory can become an issue. But I still think it can be done faster, using a double loop in a scripting/programming language, rather than shell. My perl skills are not very developed, but you can try this out:

Code:

#!/usr/bin/perl -w

use strict;

while (<>) {  
    chomp;
    my $pattern = $_;

    open FILE2, "-|", "gunzip -c file2.gz" or die $!;
    while (<FILE2>) {
	if (/$pattern/) {
	    print;
	}
    }
    close FILE2;
}

Save it as grep2.pl, make it executable

Code:

chmod u+x grep2.pl

and run it as:

Code:

gunzip -c file1.gz | ./grep2.pl

or

Code:

gunzip -c file1.gz | ./grep2.pl | gzip -c > output.gz

once you tested it out and want the output gzipped.