Shell Programming and Scripting

HI,

I have a file(suppose 1 million Records), i wanted to divide file into chunks(100 small files), these all files should run parllell(e.x like threads) to utilize the process for fast processing.

Can you please provide a solution for the above issue.

Ex:

Code:

 Split -b 2m file.txt file, filea, fileb, filec...

Required to read all the above created chunk files(filea fileb, filec ...) parllell.

Thanks,
Praveen

I do not understand what you're trying to do.

Are you saying that you want 100 copies of split running at the same to split a single file into 100 chunks? That is not the way split works! And, the split utility takes at most two file operands; not an unlimited number as indicated by the filec... in your example split command (and the comma in filea, will be used as the base part of the names of your output files (the first four of which will be named filea,aa, filea,ab, filea,ac, and filea,ad) assuming you remove the remaining file operands which would cause split
to fail with a syntax error.

You say your input file has about 1 million records. Are the records fixed length or variable length? Are you absolutely positive that files of 2,097,152 bytes will contain complete records (i.e., that the start of a record won't be in one of your split files and the end of that record won't be at the start of the next split file)?

After you have created a bunch of files (all but the last of which contain exactly 2,097,152 bytes), what do you want to do with these files? You say you want to run them. Are each of these 2,097,152 byte files a shell script???

Does your system have 100 CPUs? If not, trying to run 100 jobs in parallel will slow down processing; not provide faster processing!

Are you going to distribute the output files from split onto different disk drives? Do you have 100 different disk drives? Will running these 100 files in parallel overwhelm your I/O subsystem?

What operating system are you using?

What shell are you using?