Do we have to loop?

HTH

I'd probably use diff too...

If the lines in the files are similar to the lines you put in your first post, meaning there are no spaces on the lines, you could:

the -m1 will cause grep to exit after the first match is found. If no match is found grep will exit with status 1, you can use that to determine if the line exists in file 2 or not. Keep in mind, that "for k in `cat`" stuff will break if you have spaces in the lines in the file.

grep -v -f file1 file2

This will give you all the lines in file2 which are not in file1

Hi.
By chance I am working with a text file of this size ( 1 GB ). It contains just over 1 GB, and has 15 M (15,000,000) lines. The real time to count the lines with wc is 15-20 seconds ( AMD-64/3000, SATA disk).

If this is correct, and you have 2 such files, then I think any method that reads a line from file1 and uses it with a program to look through file 2 at each step will not end quickly, because there will be 15 M loads of that program involved, not to mention actually reading the file. For example, doing a grep reading /dev/null for 15,000 times takes about 10 seconds (10.2 actually) real time. For 1,000 times that, I'd be looking at 2.75 hours just to load grep from the disk and read an immediate EOF. A grep of a non-existent string takes about 18 seconds for a single search.

I suggest that the files be sorted and diff be run once on the two files (post #8, rikxik). That will be 2 passes across each file, a decrease of close to 100% from 15M passes over 1 file.

If my facts are wrong, then tell me where I missed something of importance or made a mistake. Otherwise, perhaps we should take a step back and you tell us what the higher purpose of the problem is -- what problem you are really trying to solve -- perhaps we can suggest some other approach ... cheers, drl

Hi drl - I was wondering whether there is any reason/performance gain (for diff) if we sort the files? Is it essential/necessary? Just thinking aloud.

Hi, rikxik.

I was thinking that the diff window to look for sequences would not be so large. However, if the files were very similar, then the sort could perhaps be skipped -- I hope for the best, but expect the worst

It would be interesting to try it both ways, of course ... cheers, drl

I did sort both the files and then tried diff as well as grep -v -f file1 file2, same problem.
It is running for too long.