Status code checker for 1300 URLs is running 15 mins

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Status code checker for 1300 URLs is running 15 mins
# 1  
Old 09-14-2011
Status code checker for 1300 URLs is running 15 mins

hi All,

I wrote a shell script for checking the status code of the pages. for 1300 URLs, it is taking 15 mins to run. Can anyone pls provide a solution to make the script to run faster. about 5 mins..

In my script i used "while loop" and "wget --server-response".

Thank You!
# 2  
Old 09-14-2011
Without knowing the script it would be hard to tell. And 1300 URLs in 15 minutes already is pretty fast for me ( 15 minutes * 60 / 1300 = ~0.7 seconds per URL, including starting the process and establishing the connection)
# 3  
Old 09-14-2011
you could try to get (only) the http status code by
curl -o /dev/null --silent --write-out '%{http_code}' <yourURL>

however I am not sure whether it will help you check 1300 urls faster. You can give it a try.
# 4  
Old 09-14-2011
Sure thanks.. i will try it..

Is there any way of running the script in parallel.. i.e.., executing the loop in parallel?

# 5  
Old 09-14-2011
Yes. As for how, we'd have to know what you're doing with the status code afterwards.
# 6  
Old 09-15-2011
Just the script have to generate the status codes for 1300 URLs, Am going to Integrate this script with a Build process. This Build process will take 40 mins to run. After this build process gets over, this status code checker will be triggered, so if this script runs for 5 mins then it will useful.

So if the script runs in such a way that 10 instances taking 10 URLs at a time (i.e., in Parallel) then the script will run in 5 mins.. Any ideas of how to do this in parallel ?

# 7  
Old 09-15-2011
Did you try with perl ?

use strict;
use warnings;
use LWP::Simple;
my @sites=qw(;
foreach my $site (@sites){
 my $filename=$site;
 my $response = getstore($site,$filename);
 if($response eq RC_OK){
  print "$site is up.\n";
  print "Theres some error on the site. Returned error code $response.\n";

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

FIFO Status Checking/Running

May i please know 1) How to check if FIFO process is down? 2) How to bring FIFO up and running? we are using redhat Linux OS and bash shell Thank you. (1 Reply)
Discussion started by: Ariean
1 Replies

2. Linux

Check up the status of a Script (running or not)

Hello, i allready search on google und here in the local Forum, but can't found something. I need a query in php, that check whether a process (script) is running or not. Like this: php query: /usr/bin/Script01 >> if runnig, then: "Script01 is Online", if not "Script01 is Offline" I... (2 Replies)
Discussion started by: ProTechEx
2 Replies

3. Shell Programming and Scripting

Get status of dd running in background job

Hello everyone While working on TUI for scripts, there there came the idea to' add a command' for dd too. That was, after 'wrapping' tar and wget either, to display their growing size and return the exit code with a textual-visual-feedback to the user. Now displaying the filesize of a... (13 Replies)
Discussion started by: sea
13 Replies

4. UNIX for Dummies Questions & Answers

302 server status code to 301/404 server status code

Hello, Sorry for my english. I have an arcade site. If database has the game name is good. if database has not the game name redirect to by 302 error code. if database has not the game name i want a 301/404 error code and no... (0 Replies)
Discussion started by: hoo
0 Replies

5. Shell Programming and Scripting

Extract URLs from HTML code using sed

Hello, i try to extract urls from google-search-results, but i have problem with sed filtering of html-code. what i wont is just list of urls thay apears between ........<p><a href=" and next following " in html code. here is my code, i use wget and pipelines to filtering. wget works, but... (13 Replies)
Discussion started by: L0rd
13 Replies

6. Solaris

Port status/Running daemon

Hi, I need to run an application (Hudson) listening to port 8080 on a remote Solaris server. I have managed to start that application and tried to access it with my browser from my local PC, but unsuccessfully. I need to find out what is blocking the access to that port (or any other). A... (9 Replies)
Discussion started by: JVerstry
9 Replies

7. Shell Programming and Scripting

How to know the status of process running in background

I have run one shell script in background that contains a endless while loop. I am not able to know the status of that job . Please provide any command to know this. I have already used "ps -aef" , "jobs" to know it , but it didn't work. I am sure the process is running as it is generating a file... (8 Replies)
Discussion started by: sumanta
8 Replies

8. UNIX for Dummies Questions & Answers

status of the script running under a particular owner?

How to track the status of a shell script running under a particular owner which is running in the background? (2 Replies)
Discussion started by: ishmael^soyuz
2 Replies

9. UNIX for Dummies Questions & Answers

Script to kill rsh processes running for more than 10 mins

Hi Friends, I need to write a script to kill some processes running for more than 10 minutes. Can I get some pointers on that. Thanks for ur help in Advance. Thanks&Regards, Amit (3 Replies)
Discussion started by: amitsayshii
3 Replies

10. Shell Programming and Scripting

Hw to Know the status of running JoB

Hi all, I am running a job .. and i want to know the status tht it is runnig or not .. and how can i find the jobId of my job .. I have to get it to kill my running job Pls let me know da Unix commands to do it .. i m wrking on Hp UNIX (1 Reply)
Discussion started by: ravi.sadani19
1 Replies
Login or Register to Ask a Question