Shell Programming and Scripting

Well yes! the file has over 1 million lines and its pretty big!

Code:

awk 'NR==FNR{A[$1]=1;next}!A[$1]' excludefile inputfile > outputfile

this command did something but it didn't work! it took only a second to run! and produced a huge file but it doesn't seem that it read the arguments from the exclude file properly.
as when I do grep for one of the numbers in the exclude file I still see it in the outputfile.

any other things I could tweak?

Thank you

Define 'pretty big'. Megabytes? Gigabytes? Terabytes? When you start processing hundreds of megs of data and up, things can take a while. You may just need to be patient. If your exclude file is also enormous, beware -- the entire thing has to be loaded into memory for your file to be processed efficiently! How much RAM do you have?

The 'grep' version ought to be particularly efficient unless you're running out of memory and eating into swap.

Could you post a sample of your input file?

These text files weren't edited/created on Windows, were they? That will throw lots of things out of whack.

Whenever awk misbehaves, try 'nawk'. Many systems use the more modern nawk by default when you run awk, a few still don't.

Could you post a sample of both the input file and the exclude file?