Hi all,

Thanks for the responses ...

I have accomplished this with the following methods, however I have gone a step further with a third method...

Hopefully they help the next one in need!
The 3rd one is pretty specific to my needs.

Method 1 based on Removing Lines if value exist in first file post #4 by Azhrei, Thanks Azhrei

Code:

perl -i~ -e '  # -i~ for in-place editing with tilde backup file
  use strict;
  use warnings;
  my @a;
  my %excludehash;
  my $file = shift;
  open(excludelist, "< $file") or die;
  chomp( @a=<excludelist> );
  close(excludelist);
  @excludehash{@a}=@a;
  {
    local($/) = "";
    while (<>) {
      m/YOURKEY:\s+(.*)$/m;
      print unless exists $excludehash{ $1 }
    }
  }' "excludefile" "datafile"

Method 2 my own brew.

Code:

perl -i~ -e '  # -i~ for in-place editing with tilde backup file
  use strict;
  use warnings;
  my %excludehash;
  my $file = shift;
  open(my $excludelist, "<", $file) or die;
  while(<$excludelist>) {
    chomp;
    next if /^$/;
    $excludehash{ $_ } = $_;
  }
  close($excludelist);
  {
    local($/) = "";
    while (<>) {
      next if ( m/^YOURKEY:\s+(.*)$/m && $excludehash{ $1 } );
      print
    }
  }' "excludefile" "datafile"

Just for giggles I created a dummy datafile that was ~26M with 80,743 records ... each record consisted of at least 7 and up to 30 lines of text. After generating an excludefile of 20,271 records to be removed... I ran them both across it.

The speed is freaking incredible!
I didn't accurately time them, however it is done in less than 15 seconds! I was/am blown away by that!
Especially on my FBSD7.1R PIII-866!

Now from the results of that education I did the following!

What is in production for my needs...
Take a look at the sample data above and you will notice that Archive Maintenance removes old archives and logs them as "Removed archive: ... ..." ... There is the EXCLUDELIST!!

The following code reads the log file in the first while loop adding all the "Removed archive:" elements to a hash (removehash).
Then moves the file pointer back to the beginning of the log, and in the second while loop scrolls down though the records matching the removehash elements to the "Using archive: of each record... If there is a match skip it!

It's even wise to multiple 'Removed archive:' elements per record...
And is incredibly fast.

Code:

perl -i~ -e '  # -i~ for in-place editing with tilde backup file
    my %removehash;
    {
      local($/) = "";
      {
        while(<>) {
          while (m/^ Removed archive:\s+(.*)$/mg) {
            $removehash{ $1 } = $1;
          }
          last if (eof)
        }
      }
      seek(ARGV, 0, 0);
      {
        while (<>) {
          m/^ Using archive:\s+(.*)$/m;
          print unless exists $removehash{ $1 }
        }
      }
    }' "logfile"

-Enjoy
fh : )_~