Shell Programming and Scripting

I've got file A with (say) 1M lines in it ... ascii text, space delimited ...

I've got file B with (say) 10M lines in it ... same structure.

I want to remove any lines from A that appear (identically) in B and print the remaining (say) 900K lines. (And I want to do it in zero time of course!)

Best I've come up with so far is somehow marking the lines in A, then doing a sort and applying an awk script to the result so that the marked lines are only printed if the following (or previous) line isn't "identical" except for the mark.

But after 1000 years of shell programming I've GOT to believe I'm missing an easier/faster solution ... I'm using bash and cygwin tools - and compiling is not an option.

ADVthanksANCE for your help!
=Gneen

Code:

cat fileA | while read line
do
grep -q "$line" fileB
if [ $? -eq 1 ]; then
echo "$line" > fileC
fi
done

Not sure how fast that would be, but fileC will end up with all the lines that were in fileA that were in not in fileB.

Heh - the grep inside the read loop would "work" ... but I'd have to come back in a year to see the results!

For tiny files this would clearly be the way to go - but for files the size I'm dealing with this would mean one million greps into a file that was ten million lines long ... can you spell "Rip Van Winkle"?


=Gneen

How about?
This would effectively break up everything into 26 smaller files based on the first character of the file, and assuming it is lowercase. (Or, depending on the format of your data, could be ten numeric groups, etc...)


for outch in a b c d e f g h i j k l m n o p q r s t u v w x y z
do
cat fileb | grep ^"$outch" > fileb_"$outc"
done

while read zf
do
leadch=$(echo $zf | cut -c1-1)

now do lookup to appropriate file
use the just determined $leadch variable
and write if found/not found, as you like

done <filea

Assuming awk is fairly memory efficient and you have at least 1M x length-of-line bytes in virtual mem, this should work:

Code:

awk 'NR==FNR { A[$0]=1; next; } { if ($0 in A) { A[$0]=0; } END { for (k in A) { if (A[k]==1) { print A[k]; } } }'  A   B

Thanks otheus!
Nothing quite like a one-line cryptic awk script from a guru ... with a few minor typo corrections it shows excellent promise ... trying it with the giant files and the real data is going to need to wait for tomorrow. SWEET! (I'll post back here with some timing results.)

And thanks to to the other folks who replied - this is indeed an incredible resource!

Code:

grep -v -f fileA fileB > output.txt