Shell Programming and Scripting

not sure but i guess it can handle any number of record.
may be other can confirm us.

comm compares the entire line, while the original post mentions using only the first 3 fields. Perhaps a sort prior to comm would be in order:

Code:

sort -u -k 1,3

Although the following AWK may suffice:

Code:

awk 'FNR==NR {a[$1$2$3]=$0} FNR!=NR {delete a[$1$2$3]} END {for (i in a) print a[i]}' Old_file.txt New_file.txt

Regards,
alister

Hi ahmad.diab,
no, I didn't insinuate that awk and siblings would barf on big files, or big records. I only expressed my uncertainty whether comm would be sufficient for the job on big files.


Best regards,
Lakris

Two points:

1. comm requires the input to be sorted.
2. awk will be slower compared to comm

You can try:

Code:

grep -vf new_file old_file

I don't think this will scale either,

with more number of patterns to be searched and more number of input records, the search is almost, o(n2)

and that is not the case with awk where the power of associative arrays can be used.

Build a map, and check if the map as you iterate through each records but the check is that you don't which file to use to create a map as we don't know the number of records before processing.