Shell Programming and Scripting

Hi,

Could someone please help me with the best approach to compare lines from one file to another? Here is how I have entries -

File 1

Code:

a1
a2
a3
a4
a9
a10
a15

File2

Code:

a5
a6
a15
a7
a9

Expected output is -
a15
a9
(order doesn't matter)


The file1 is a huge file. I am trying to execute following command -

Code:

grep -f File1 File2

When I do that, I get the error after a minute- grep: Memory exhausted.

Strangely, when I do grep -f File2 File1, then it doesn't give that error but output it generates is wrong as it prints some output that is not in File2!

Please help with what is the better way to compare and print output?

what do you mean compare? you could use "diff" but that will only tell you the difference between the files. What are you ultimately trying to do?

How huge is file1? If it can't load all the strings into memory, then it just won't go. You may have to process it in smaller batches then merge them.

I can't use diff. I need to find those entries in File2 that are present in File1. They may not be in the same order. Please check expected output I have provided and input files.

Thanks for your help.

---------- Post updated at 01:12 PM ---------- Previous update was at 01:11 PM ----------

Corona,

That is also true. Would you know any command that splits a file into multiple chunks?

Thanks,

Yes, 'split < File1' would produce files xaa, xab, ... in chunks of 1000 lines. The parameter '-l 10000' would make it split at 10000 lines. This could produce hundreds or thousands of files if File1 is enormous enough, be warned.

Without knowing the sizes of files involved it's not possible to give a precise answer.

this should work if file sizes are not too extreme; what is aproximate file sizes?

Code:

# first files need to be sorted
sort file1 > file1_sorted
sort file2 > file2_sorted

comm -1 -2 file1_sorted file2_sorted

the limited example given this works. If too large as stated above files will need to be split first.

the below link references a script that would breakup your files into whatever size chunks you specify.