Programming

I have an extractfile (with fields delimited by pipes '|') and I want to prepend a counter based on the below requirements:
- The counter starts at 3.
- The counter increments only if the date (67th field of the extractfile) is different.

Below is what I started off with:
$cnt=2;
$date="dummy";
open( extractFile, "<$extractFile" ) or die "ERROR: Can't open file: $extractFile - $!\n";
open( outputFile, ">$outputFile" ) or die "ERROR: Can't write file: $outputFile - $!\n";
while( $extractLine = <extractFile> ) {
# if the date is the same then prepend the counter without incrementing it
print outputFile "$cnt|$extractLine";
# else if the date is different, then increment the counter and then prepend it
cnt++;
print outputFile "$cnt|$extractLine";
}

I want to avoid using the 'split' by pipes to read in the records because I will be dealing with 800 million record extract. So I want to make this process as efficient as possible.

To clarify, I need help in reading in a certain field (which is a date - 67th field) from a file without using the 'split' command.

Thanks,

- CB

awk splits fields by default, so it is as good as it gets at doing that: try this

Code:

awk -F'|' ' BEGIN{cnt=3; old=""}
      {if (old != $67)
          { old=$67
            printf("%d|", cnt++)
          }
        print $0
       } ' inputfile > outputfile

But I don't understand why perl's split is off the menu. Did you try something to prove it doesn't work well?

split is not very efficient in splitting 800 million records each having 100 fields.

- CB

I think everyone would agree that splitting 800 million records is never a fast task. split might not be efficient, but regex sure are slower.

I would advise you to try and code a C app for that.