Shell Programming and Scripting

Hi All

I have a somewhat complex requirement.
I have a file containing about 1 million records
The records in the file are of fixed length
Every record begins with 03,04 ,05 or 06

03 record is parent record.
04 05 and 06 are child records
Every 03 record can have
zero ,1 or more than 1 child record.

The position 12:16 of 03 record indicates the TYPE .Every record has a TYPE There are about
10 unique file TYPEs in the whole file


Now my requirement is that
IF number of 03 records for a particular TYPE is less than 100 ,
then ALL the 03 records for that TYPE along with their CHILD records should be written to a new file.

IF number of 03 records for a particular TYPE is more than 100 ,
then ONLY 2% of the 03 records(selected randomly) for that TYPE along with their corresponding CHILD records should be written to a new file.


Please note that all the records need to go to a single file.

Is it somehow possible to achieve this using shell scripts.

Thanks in advance
Ashwin N.



Thanks
Ashwin N.

If I had to do this in a shell script I would use the awk command.

Yes. But how do I get the lines in the file
randomly.
I have worked on few simple shell scripts
but never really used sed and awk

Thanks
Ashwin N.

The HP-UX 11.00 awk documents random functions that can be used within an awk script.

Run the command 'man awk' to get a description of what awk can do.

I think that I would do two passes of the data file. First to find all types that occur more that 100 times...

Code:

awk '
  /^03/ { a[substr($0,12,5)]++ }
  END { for(i in a) if(a[i]>100) print i }
  ' data_file > tmp_file

And then a second pass to filter out data records...

Code:

awk '
  BEGIN { srand; while(getline < "tmp_file") a[i++]=$1 }
  /^03/ { f=0; for(i in a) if(substr($0,12,5)==a[i]) f=1 }
  /^03/ || f==0 || f==1 && rand*100<2
  ' data_file > new_data_file

Hope that this will get you programming in awk.

This can be done in most languages. I would go with c mostly based on file size. If not c, then I would use ksh. The choice of language really is just a personal choice.

This task is tough enough that it will require a real programmer. You say that you have worked on a few simple shell scripts. Unless you have several years of programming experience in some procedural language it's unlikely that you will succeed.

To answer your question, you need to make a pass though the file creating lists of each of the types of 03 records. Then you need to examine the lists. If a list has less then 100 items, all items on the list will be marked ok. If the list has over 100 items, only some will be marked ok.

Since you counted the elements, you can compute 2% of that number. Now you know how many to mark ok.

To mark one ok, generate a random integer between 1 and n where n in the number of non-ok elements in the list. Now scan the list from element 1 to the end and find the selected element. Mark it ok. Subtract one from your number of non-ok elements on the list. Generate a new random integer between 1 and current number of non-ok elements. Find that one. And so on until you have marked the required number ok.

After you have done this with each list, go back to main file. For each 03 record, see it it's marked ok on the lists. If so write it and the child records to the final output file.