Shell Programming and Scripting

This updated version checks for different lines that have the same MD5 and fails with an exit code so at least you can be confident nothing is wrong. This needs a filename parameter as it needs random access and streams don't support that.

Code:

#!/usr/bin/perl -w
use Digest::MD5 qw(md5);
open(DATA, "<", $ARGV[0]) || die "Can't open $ARGV[0]: $!";

my %seen;
while (1) {
  my $pos = tell DATA;
  my $line=<DATA>;
  last unless defined $line;
  my $crc= md5($line);
  if(!$seen{$crc}) {
     $seen{$crc}=$pos+1;
         print $line;
  } else {
      $pos = tell DATA;
      seek DATA, $seen{$crc}-1, 0;
          $!=255;
          die "Duplicate CRC found" unless (<DATA> eq $line);
          seek DATA, $pos, 0;
  }
}

Takes 110 seconds on the already duplicate line removed database. Size and wc -l match.

I will have to build the raw database again (~70 min) to see if there are collisions. I am leaving work in a few min and will let it run when I am gone.


Mike

It's really just for you piece of mind as the chance of two different lines having the same MD5 is so low it should never happen.

And they did not happen Just curious (as I don't know any Perl) does the script complete the output file and give the error messages or does it terminate at the error?

Mike

You'll have to be more specific. It's not black and white.

It terminates setting the error num $? to 255. You will get the partial file up to the error output.

The database in now up to about 4GB. Chubler_XL's duplicate line removal script is still working great. I now build my database in segments and concatenate and I accidentally has more than one header in there which tested how the script handles a crc collision (terminated and output file is empty except for header).

This got me thinking about the possibility of instead of terminating, reverting to testing the entire line's text just in the event of the incredibly rare crc collision?

Mike