Quote:
Originally Posted by
uiop44
I have two very large datasets (>100MB) in a simple vertical list format. They are of different size and with different order and formatting (e.g. whitespace and some other minor cruft that would thwart easy regex).
Let's call them set1 and set2.
I want to check set2 to see if it contains any of the data entries in set1. I think of this as individual greps of set2 using each line of set1.
(NB- I could, with some work, manipulate the sets to make the order and formatting the same.)
In your opinion, what is the best tool to use for this search of set2 using the data in set1?
- comm?
- a looping shell script, or xargs, that calls grep?
- grep -f?
- diff?
- combine the sets (after making format the same) then sort and print only duplicate lines? uniq -d, sed or awk
I would implement them like this ->
1. sort both the datasets (presorting always helps for better performance especially if the datasets needs to be updated frequently and the operation is carried out many times.
2.Lets say set1 has index1 and set2 has index2
3.add a very large element say 10000000(which is larger than the largest of the elements in set1 and set2) to set1. See below why.
4. while not(end of set2) // set2 is the bottleneck
{ print set2[index2] if set1[index1]==set2[index2];
index2++ if(set2[index2]<set1[index1])
index1++ if(set2[index1]>set1[index1])
}
P.S . we added it to exhaust set2 which is the condition of the while loop. However you could have done while not(end of set1) And not(end of set2) {} but since you only wanna check set2 so we had to stick to above to make it more efficient.
enjoy,
Gaurav