Shell Programming and Scripting

I have a 2 files in .gz format and it consists of 5 million lines the format of the file would be

gzcat file1.gz | more

abcde
aerere
ffgh56
..
..
12345

gzcat file2.gz | more

abcde , 12345 , 67890,
ffgh56 , 45623 ,12334


whatever the string is in the file1 should be matched with the file 2 and the complete line in file 2 need to be printed in file 3

if it can executed with the zip format only it would be helpful

tried fgrep it is failing due to source file due to memory limitations

So need some script using PERL or AWK.

Would this get you what you want?

Code:

gzcat file1.gz > file1.txt
gzcat file2.gz | grep -f file1.txt - > allLinesMatched.txt

awk in bash/ksh93 shell:

Code:

awk -F '[ \t,]*' 'NR==FNR{A[$1];next}{for (i=1;i<=NF;i++) if ($i in A) {print;next}}' <(gzcat file1.gz) <(gzcat file2.gz) | gzip > outfile.gz

awk only:

Code:

awk -F '[ \t,]*' -v if1=file1.gz -v if2=file2.gz -v of=outfile.gz  'BEGIN{
  cmd="gzip>"of
  while("gzcat "if1 | getline){A[$1]}
  while("gzcat "if2 | getline){for (i=1;i<=NF;i++) if ($i in A) {print | cmd ;i=NF+1}}
}'

It is giving a memory limitation error @ mrini