Shell Programming and Scripting

Hi.

If you want to use the single-process perl code as is, you could make use of parallel. From the GNU project page in part:
From my point of view, you'd split the input file of URLs into smaller files, then feed each one into a process generated by parallel. You could arrange to skip lines in the same input file, but that design seems error-prone.

Not knowing any more about the process except that it uses curl, it would seem to be network-bound, so it's unclear if anything is gained by parallelism, and, in fact, additional overhead might make the situation worse. Seems like a good example for a benchmark.

Best wishes ... cheers, drl

Thanks for your input. I've tried running the script in parallel, but that dosen't seem to be more productive. Right now the existing script takes 6 hours to complete the whole run, I wanted it to perform better. This is the reason I was thinking about running it in multiple threads.
Example: The input URL text file has 100 urls, My script should in 5 threads hitting 5 urls at one time. Which means it will run 5 times faster.

I want to make sure if this is possible in Perl scripting or Should I start looking for other options.

Thanks In advance.

It's the same idea. Each seperate process is independent -- even more profoundly independent than a thread. Yes, this is how parallel works too. The idea would be to split the list in 5 and run 5 instances, each would process their chunk of the list independently. If it didn't work then there's a few possible reasons. Are you sure you only gave each instance part of the list instead of the whole list? Are you sure your processes were in the background instead of waiting for one to finish before starting the next? Or your connection may simply be throttling you from opening too many connections too fast...

So: How exactly were you running instances in parallel?