Shell Programming and Scripting

Not looking into the aspect whether the above code solves OP's question or not..
this is UUOC ..

the use of 'cat' in the above command is not needed

Another short cut,

use List::Compare - answer is available in a single command and there are multiple options associated with that

Hi prvnrk,

Code:

#for "numbers existing in both files"
fgrep -f file2 file1

#for "existing in file1 but NOT in file2"
fgrep -v -f file2 file1

This will be one of the fastest solution in unix/linux (because you mentioned "efficient search"). Use fgrep instead of grep here and 200K*2 lines it will just finish in few seconds.

meharo

Indeed, if you could use hashing in perl, by loading 'hay' to memory and searching 'needle' from that, it should be much more faster than that.

try fgrep -v -f file1 file2

this should be very quick

shell> cat file1
111
222
333
444
shell> cat file2
222
444
666
888
shell> cat file1 file2 | sort | uniq -d
222
444
shell> cat file1 file2 file2 | sort | uniq -u
111
333

here is my small test time for taking grep method and my method
shell>wc -l file*
28684 file1
4993 file2
shell> time grep -f file1 file2 |wc -l
4953

real 5m34.005s
user 5m26.267s
sys 0m0.785s

shell> time sort file1 file2 | uniq -d | wc -l
4953

real 0m0.224s
user 0m0.218s
sys 0m0.006s
shell> time comm file1 file2 -12 | wc -l
4953

real 0m0.005s
user 0m0.004s
sys 0m0.001s


all three are givig same output
method 1:-comm is efficient but if input files are not in sorting order this method wont work you need to create temporary files for both file1 and file2. If you create temperary sorted file1.sort and file2.sort it takes same time as sort&uniq method
method 2:- grep method is not efficient its taking 5min 34sec you use this method if file1 contains 2 or more feilds and file2 contains one feild cat file1 | grep -f file2
I thik grep process like it will read one line from file2 and grep the line from file1
while read line
do
grep $line file1
done<file2