Shell Programming and Scripting

Hello Friends,

I have the below scenario in my current project. Suggest me which tool ( perl,python etc) is best to this scenario. Or should I go for Programming language ( C/Java )..

(1) I will be having a very big file ( information about 200million subscribers will be stored in it ). This is a static data and will be getting changed once in a month. fields in this file will be like : AAA, BBB , CCC

(2)I have to process input data ( number of records / hour could be around 100 million records ) ( fields : AAA , XXX , YYY , ZZZ etc ).
A lookup needs to be made for each field in input file with file in step(1) and produce output : AAA , XXX , YYY , ZZZ , CCC. ( Lookup will be based on field "AAA" ).

Any suggestion on:
how to process each and every input record against such big static file?

Regards,
Ravi

The very best tool for this is a database application - mysql, oracle, etc. Create an indexed table from your "big file", update it once a month. You gain scalability, meaning you can write one small db app, and run many separate parallel processes. Or threads.

Otherwise you would need a hash of 200 million records to do real time lookups. Not that this is not possible, it just seems like an unstable or error prone approach to me.
Plus it may not scale well as load increases.

So, with no database you need major hash support in your app- and tons of free memory

Code:

200 million * [big file record size]

probably way more 4GB.

perl, ruby, C will work either with or without a db. Shell/awk will not work at all well.

Hi Jim,

Thanks for the suggestion.

I got your point and was thinking in the same way.

However, If I store the "static data" in DB,

While processing each and every record ( input dynamic data which is around 100 million / hour ) , I have to do a DB lookup for each and every record...!

Is not that expensive?

I agree with Jim's point of vue, I am lucky, in such case I go see my firends one level lower, since I am responsible of their architecture I do ahve some favors when needed: I use SAS... but SAS cost is $$$
What is the expected file size?

Hi Vbe ,

Not sure which file you are referring to here: Expected file size of?

The static file will be having 200 million records with record having around characters in it approx. This file i can store in DB which is a one time task ( once in a month ofcourse).

Now,my worry is , I have to do a DB look up for each and every input record I receive and extract some value from "DB" and do the changes in input record and produce the output.

The input record will be around 200~250 bytes in length and approx 100 million records needs to be processed per hour.

Any suggestions?

Regards,
Ravi

Certainly less expensive than looking up records in a flat file!

Or do you mean that you coudn't just "do x for all records" in a database? Because actually, you can.