UNIX for Dummies Questions & Answers

New to unix. I have a couple files of 5 million records. I have a key field on those records. I have about 300 keys that I need to remove off the file, and I don't want to write a program to do it. I have used grep -v in the past and that works great for a few records, but I can't see myself having to do that 300 times/file.

Is there an easier way using grep, egrep, sed/awk, etc.... that I remove this records quickly. The file layout is simple:

H00012345

The key starts in position 5. of the file. In this example, I would need to remove 12345 from the file of 5 million records. My problems is that I have 300 differenet keys/records that need to be removed. I know what each of the key values are, but I don't want to have to remove them one at a time.

Example

H00011111
H00012345
H00022222
H00033333
H00044444

I need to remove H00012345 from the file and have the following result.

H00011111
H00022222
H00033333
H00044444

Thanks.

Put the three hundred items in file1. assume the big file is called file2

Code:

awk 'FILENAME=="file1" {arr[substr($0,5,5)]++}
       FILENAME=="file2" {t=substr($0,5,5); if(t in arr){next} else {print $0}}' file1 file2 > newfile

Thoe order of "file1 file2" on the command line is very important. Has to be as given.

Here is a solution using grep. It involves using -f option to read patterns stored in a file (key_file) and using regular expression to search starting at the 5th position.

Code:

cat key_file
^....12345

Code:

grep -vf key_file master_file > new_file