Shell Programming and Scripting

hi All,

I wrote a shell script for checking the status code of the pages. for 1300 URLs, it is taking 15 mins to run. Can anyone pls provide a solution to make the script to run faster. about 5 mins..

In my script i used "while loop" and "wget --server-response".

Thank You!

Without knowing the script it would be hard to tell. And 1300 URLs in 15 minutes already is pretty fast for me ( 15 minutes * 60 / 1300 = ~0.7 seconds per URL, including starting the process and establishing the connection)

you could try to get (only) the http status code by

Code:

curl -o /dev/null --silent --write-out '%{http_code}' <yourURL>

however I am not sure whether it will help you check 1300 urls faster. You can give it a try.

Sure thanks.. i will try it..

Is there any way of running the script in parallel.. i.e.., executing the loop in parallel?

Thanks

Yes. As for how, we'd have to know what you're doing with the status code afterwards.

Just the script have to generate the status codes for 1300 URLs, Am going to Integrate this script with a Build process. This Build process will take 40 mins to run. After this build process gets over, this status code checker will be triggered, so if this script runs for 5 mins then it will useful.

So if the script runs in such a way that 10 instances taking 10 URLs at a time (i.e., in Parallel) then the script will run in 5 mins.. Any ideas of how to do this in parallel ?

Thanks!

Did you try with perl ?

Code:

 
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
my @sites=qw(www.microsoft.com www.google.com www.yahoo.com);
foreach my $site (@sites){
 my $filename=$site;
 $site='http://'.$site;
 my $response = getstore($site,$filename);
 if($response eq RC_OK){
  print "$site is up.\n";
 }
 else{
  print "Theres some error on the site. Returned error code $response.\n";
 }
}