Shell Programming and Scripting

CRGreathouse,

Thank you for C

I had been using shell scripts because I could show results quickly and then - gee, guess what - file sizes got bigger. May have to implement with C. Tenfold, at least, improvement is significant.

Big problem - that is very hard to describe - I have no compiler at work - not approved - really, really long story, but simple shell scripts can be implemented so, I have gone that route.

Thanks again tho, speed improvements can make a case for more tools, but bureaucracy can be stifling.

macsurveyr

No problem. If the biggest you're going to do is 500x5000, then vidyadhar85's script should be fine. If you needed to do 50,000x50,000 then it would take ~4 hours, at which point it may be worthwhile to ask for an exception or find a workaround. (If you have chmod, you could compile elsewhere and transfer in some form like base64...)

Speedups (@ 5000x5000):
Original: estimated 30 minutes
vidyadhar85: 2.2 minutes
my executable: 12 seconds
optimized* executable: 2-3 seconds?
multithreaded** optimized executable: < 1 second?

* The primary optimization here would be avoiding temporaries.
** 4-way on a quad core machine producing four files that are joined at the end.

Whoa,

Time savings are impressive.

I do have a dual processor quad core machine available - 8 virtual - two software packages I use actually take advantage so I know it actually works - and your numbers show that it would be almost instantaneous compared to 30 minutes.

hmmm thank you!!!

The multicore savings are only for programs (unlike mine) designed to take advantage of it. But I think my numbers are reasonable here, depending on how much a really smart programmer could beat gcc's optimizer.