Shell Programming and Scripting

I have not assumed anything but what was stated in the original post. And yes, if there is sufficient ram to hold the data in memory, then most definitely that would be the best way to go (unless you want to wait days instead of hours).

The 600,000 figure comes from the original post, which states that the data contains 2,000 rows of 600,000 columns each (that's 600,001 awk invocations, 1 to determine the number of columns in the first row plus one per column), with each of those invocations reading the entire file.

A test run on a small (compared to 2,000x600,000) 500x1000 matrix shows that your i/o heavy approach takes 2m35s versus 5s for the ram hungry solution. So, yeah, I think it's a good idea to fill up memory when the gains are on the order of 30x faster (1 hour versus 1.25 days). And as the data set grows, so will the disparity in performance.

Obviously, if the machine does not have sufficient ram, then my approach is not feasible. If it does, it's the way to go.

Ok, so the OP has 2000x 600000 columns. that is over 2GB in file size. do you slurp more than 2GB into memory to do such stuff? What about other processes?

Hi kurumi:

What about other processes? Perhaps there are none besides the basic os services. Perhaps there are thousands. Perhaps there is a dedicated machine for this problem. Perhaps not. Perhaps the hardware available to solve this problem is an old pentium with only a few megabytes of free ram. Perhaps it's a 64-bit monster with many gigabytes of free ram. I do not know. You do not know.

What we do know is that your solution is horribly slow and inefficient even on comparitively small datasets, but it may be the best and only possible approach if there is insufficient ram. My solution is much faster, but horribly ram hungry. This type of trade-off (ram vs i/o-time) is common in algorithmic design. Choice is good and the original poster can choose which best suits his situation, if indeed either of these solutions is suitable.

If I had that 64-bit monster at my disposal, I know which approach I'd choose.

Regards,
Alister

I'd be interested in timing the various solutions on your large data.

Just to update.
I am running the solution from cjcox since 3 days.
I am almost at the end of the run.
Will test that & report.
I will also test kurumi's solution if you can re-post it.
And alister's solution too.

All I can say is thanks for the solution.. but it takes a long time !
Can you let me know how to time the run already started?
Thanks

Hello, genehunter:

I don't think you can time it after having started it, except perhaps by using ps to determine the process' start/elapsed time. You can then poll once per minute or so with a script or cronjob until the process is no longer running, at which point run date to record the current time.

For future runs, prepend 'time ' to whatever command you use to launch the job.

On a different note, I'm curious. What's the size of the data file and what are the specs of the machine being used to process it?

Regards,
Alister

The size of the datafile is 1.9G
I am running on a 12 core Xeon with openSUSE 10.2 (X86-64)
66002864k total, 65684428k used.
Hope some of that makes sense!