Shell Programming and Scripting

There's a few ways to parallelize this, but for that we'd need more info. What system are you on? Is using GNU parallel an option? How do you get the input? What do you do with the status code once you've got it? What shell are you using?

After generating the status codes, output file will be sent to some people.

Am accessing the Bash shell script in a unix server. I will access this Unix server by using putty.

GNU parallel is not installed in my unix server.

Which is the better way to parallelize ?

So the basic structure is

Code:

for each URL
  wget status code from URL >> codes_file

send codes_file through email

Does that sound about right?

This is what i used.. by using while loop and wget. This sounds like it will take the same time as my script..

Any other ideas?

My response wasn't a suggestion on how to make it faster, but a question on how you're doing it now, so that we can get an idea on what might work, and what side effects should be considered.

since you have wget, you probably have Linux, and can make xargs do this:

Code:

xargs -d '\n' -P 4 --max-args=16 wget -nv --spider <urllist >responselist

This will run four simultaneous instances of wget. The '--max-args' stops it from feeding too many args into one wget, so in case one download hangs a while, the other instances will be able to take up most of the slack.

The --spider tells it not to download the page, just check its existence, which should also help improve script speeds.

The -nv tells it to print success or failures one per line.

thanks.. i tried with GNU parallel.. it seems that the script completes in 4 mins!! I will try with xarg also..

Thanks!