Shell Programming and Scripting

I have a very large file (10,000,000 lines), that contains a sample id and a property of that sample. I have another file that contains around 1,000,000 lines with sample ids that I want to remove from the original file (create a new file without these lines).
I know how to do this in Perl, but it is too time consuming to run. I am aware of sed and awk as commands that should be able to complete this task in a much faster time. I have tried to implement codes that I thought would work, even after consulting previous posts, none seem to quite cover it. I also find it hard to debug as the server I'm working on is French so I don't understand the error messages of my command.

Please could anyone suggest a quick way of achieving this ?

Here are examples of the files I'm dealing with.

Here is a tab delineated sample id and property.

Code:

HELIUM:1:2:3      ABCDEF
HELIUM:1:2:4      ADEFBC
HELIUM:1:2:5      BDFACE
HELIUM:1:2:6      BEBACG
HELIUM:1:2:7      ABCDEF
HELIUM:1:2:8      ADEFBC
HELIUM:1:2:9      BDFACE
HELIUM:1:3:0      BEBACG

Here is a list of ids (The common prefix is missing) I wish to remove:

Code:

:1:2:3
:1:2:5
:1:2:6
:1:2:9

Many thanks in advance for any help you can provide.

Have you tried using grep? Use it as

Code:

grep -v -f "ids.txt" "sample-id-property.txt" > remainder.txt

Please check how much time its consuming.

Code:

grep -v -f ids_to_remove_file orignal_file

If your second example 1:2:3 is representative of the actual file contents of the small file, i.e., it has no prefix and no suffixed data either

Code:

awk -F':'  ' FILENAME=="smallfile" {arr[$1 $2 $3]++}
               FILENAME=="bigfile" {tmp=$2 $3 $4; if(tmp in arr) {next}; print $0 }
           ' smallfile bigfile >  newfile

Also

Code:

export LC_ALL=C

may help your error message language problem.

Great, thank you!
The grep works, but I was afraid it would also run too slowly. I just did a sample that searched 10,000 lines for 1,000 ids and it worked in about 2 seconds. I'm rather happy with that. I just hope the large files don't add too much load.

@jim mcnamara. The awk works wonderfully, but how do I get the data into a new file rather than print?

Thanks a lot for the help with the language problem. I'll definitely use that.

Add the code in red.

For comparison, the awk does the same task in 0.4 seconds.
Many thanks!