Shell Programming and Scripting

Hi Gurus,

I have two big files. I need to compare the different. currently, I am using

Code:

 
sort file1 > file1_temp; 
sort file2 > file2_tmp 
diff file1_tmp file2_tmp

I can use command

Code:

 
grep -v -f file1 file2

just wondering which way is fast to compare two big files.

Thanks in advance.

It depends on how much input data you have.

The grep method is very fast if you have enough memory, but that is its limit... If file1 is too large, it's liable to run out of memory and grind to a halt, or just plain crash. I wouldn't trust it with a file1 larger than a hundred or two megabytes. (file2 can be any size, though.) You should be doing grep -v -F -f file1 file2 by the way -- the -F makes sure the lines are all considered raw, instead of being used as regular expressions.

The sort method can reliably tolerate any size of input (though I would have used comm -1 -3 rather than diff).

So all else being equal, I'd use the sort method and worry less.

The file after the -f is read into memory, so it should be

Code:

grep -v -f smallfile bigfile

Some grep versions are rather slow on this, and even the faster full-line-match is not a race car:

Code:

grep -v -x -f smallfile bigfile

In this case consider replacing it by awk:

Code:

awk 'FILENAME=="-" {s[$0]; next} !($0 in s)' - bigfile <smallfile

You could time the two approaches using small to medium size files. The -v option shows additional info on resource usage.

Thanks all of you for your good suggestion.