Shell Programming and Scripting

In theory, this will work (especially with -F), but for very large files, I haven't tried, and I'm unsure of grep's algorithm here. If for every line in fileB it loops through every line in lineA, this will take a while. Or, if it tries to build a regexp tree for 1M entries, this will take some huge resources!

So, I made a 1M line file and 10M line file to test on... running the grep loop on a pretty beefy server (dual 2.8ghz intel xeons) and it took 170m ... so, not forever but definitely not as fast as you'd want.

Hi.
If the original order is not important you could sort both files, and use comm, where you can choose a list that contains lines unique to one file, unique to the other, common to both, or any combination of those ... cheers, drl

Just do sort both the file on the same field and on which you want the
new files should not contain the same record, if you have sorted on the
field no 1 on both the files use the following:-
now you want to remove the duplicates from the second :-

join -1 1 -2 1 (-t and delimiter) -v2 -o 1.1 1.2 1.3 1.4 file1 file2 > output

and your job is done in not time this will take the least time ??:

You can give this a try:

Code:

awk 'NR==FNR{arr[$0];next} !($0 in arr){print}' A B

Regards

I can't thank you all enough!

I ended up going with the awk script suggested by Otheus above ... I was so amazed at the speed I felt obligated to check the result with an alternative method - and the result was indeed verified.

My box has 6GB of memory BTW - but it would appear that my gawk has a 1.5GB limit (either compiled in or part of the OS - but in either event I don't think I can change it). The limit is approached when the size of FILE1 approaches 1.5GB ... for larger files I split the input file and ran it against each of the parts. The size of FILE2 does not play into the amount of memory required by the awk program. Your awk may have a different limit which you'll discover if it is an issue.

FILE1: START.100K
66,831,529 bytes with 100K lines (yes - my data is actually 600+ bytes/record)

FILE2: REF.1M
648,903,713 bytes with 1M lines - obviously similar data

real 0m23.323s !!!!
user 0m11.484s
sys 0m8.233s

AND THE OUTPUT IS:
48295948 bytes with 71,836 lines - i.e. there were 71,836 lines of the 100K lines that did NOT appear in the 1M line file.

Thanks for the benchmark, very intesting tread here. I have to work with big files too sometimes.