UNIX for Advanced & Expert Users

Yes!

This looks great! Thanks so much!

This code was giving a record for each ID plus a blank field 4 line after the records with data, so I changed

Code:

for(i = 1; i <= n; i++)

to

Code:

for(i = 1; i < n; i++)

and that seemed to give the desired results and that seemed worthwhile given millions of records.

One additional aspect is that the federal government delivers this data monthly as four files:

thisyear.txt
thismonth.txt
changes.txt
previousyears.txt

I know I could make a copy of this code for each file and change the input datafile, but I was wondering how to loop over the four input files in one program.

Thanks so much for all your help!

---------- Post updated at 12:45 PM ---------- Previous update was at 10:08 AM ----------

I got rid of the leading

Code:

awk '

and the close single quote from the last line and ran the script via

Code:

awk -f myscript.awk thismonth.txt

and that seemed to work.

Thanks!

If field 4 contains semicolon terminated fields instead of semicolon separated fields, that is a good change. If field 4 contains semicolon separated fields as shown in you sample input, this change will discard the last subfield in field 4 for each line and will completely skip lines that only have one sub-field terminated by the field separator (|). If some lines have an empty subfield after the last semicolon, you could check for an empty subfield before printing an output line.

If you want to produce one output file from the concatenated four input files, just change:

Code:

awk -f myscript.awk thismonth.txt

to:

Code:

awk -f myscript.awk thisyear.txt thismonth.txt changes.txt previousyears.txt

But, if changes.txt contains additions and deletions, you'll need to modify the script to ignore deletions and only print additions (or if you have a combined file and need to remove other lines that have already been processed).

And, if you want the script to switch output files when it starts processing a new input file, you'll also need to make some minor changes to the script for that.

If you need help with additional changes like this, give detailed descriptions of how field 4 is formatted, how output file names are related in input file names, etc. for the changes that you want to make.

Thanks so much for the helpful reply!

Yes, field 4 is terminated by

Code:

;

and not merely separated.