Shell Programming and Scripting

Fixing this data at source is not currently an option although I agree it would make more sense, I currently have to work with the data I am sent.

A typical example of the data would be

Code:

9999,20-NOV-2009,XXX,YYY,0,LF,BUN,EE,L,14,07-NOV-2009,0,1,0,0,0,0,.003
9999,26-OCT-2009,XXX,YYY,0,GU,BUN,LE,L,42,15-SEP-2009,0,1,0,0,0,.131,.131
6666,24-MAR-2010,AAA,BBB,0,BO,MUB,EE,L,1,24-MAR-2010,0,1,0,0,0,.077,.077

Note that even though each row contains 18 fields it is only the first ten fields which make up the unique key.

I am currently playing with the idea of loading the total sent rows into an oracle table and then loading each file recieved into a temp table allowing me to do some set arithmetic on it to get what I need (SELECT TABLE2 MINUS TABLE1), but if possible I would prefer to do this just using files on the server.

Thanks for your help so far ppl

One approach,

use BDB's to store unique records ( based on the set of keys ) and this BDB needed not be loaded to primary memory completely, tie it to the disk and treat as though you are working on hash ( internally it will keep flipping between using primary and secondary memory )

Code:

- before any new record is processed
- form a key from the record with data from the current file
- check if the key is there in the BDB - hash lookup
- if there, record is there already there but probably could have different values, check that
- if not totally new entry, store it as a key value pair