Status code checker for 1300 URLs is running 15 mins

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Status code checker for 1300 URLs is running 15 mins
# 8  
Old 09-15-2011
There's a few ways to parallelize this, but for that we'd need more info. What system are you on? Is using GNU parallel an option? How do you get the input? What do you do with the status code once you've got it? What shell are you using?
# 9  
Old 09-15-2011
After generating the status codes, output file will be sent to some people.

Am accessing the Bash shell script in a unix server. I will access this Unix server by using putty.

GNU parallel is not installed in my unix server.

Which is the better way to parallelize ?
# 10  
Old 09-15-2011
So the basic structure is
for each URL
  wget status code from URL >> codes_file

send codes_file through email

Does that sound about right?
# 11  
Old 09-29-2011
This is what i used.. by using while loop and wget. This sounds like it will take the same time as my script..

Any other ideas?
# 12  
Old 09-29-2011
My response wasn't a suggestion on how to make it faster, but a question on how you're doing it now, so that we can get an idea on what might work, and what side effects should be considered.
# 13  
Old 09-30-2011
since you have wget, you probably have Linux, and can make xargs do this:

xargs -d '\n' -P 4 --max-args=16 wget -nv --spider <urllist >responselist

This will run four simultaneous instances of wget. The '--max-args' stops it from feeding too many args into one wget, so in case one download hangs a while, the other instances will be able to take up most of the slack.

The --spider tells it not to download the page, just check its existence, which should also help improve script speeds.

The -nv tells it to print success or failures one per line.
# 14  
Old 09-30-2011
thanks.. i tried with GNU parallel.. it seems that the script completes in 4 mins!! I will try with xarg also..

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

FIFO Status Checking/Running

May i please know 1) How to check if FIFO process is down? 2) How to bring FIFO up and running? we are using redhat Linux OS and bash shell Thank you. (1 Reply)
Discussion started by: Ariean
1 Replies

2. Linux

Check up the status of a Script (running or not)

Hello, i allready search on google und here in the local Forum, but can't found something. I need a query in php, that check whether a process (script) is running or not. Like this: php query: /usr/bin/Script01 >> if runnig, then: "Script01 is Online", if not "Script01 is Offline" I... (2 Replies)
Discussion started by: ProTechEx
2 Replies

3. Shell Programming and Scripting

Get status of dd running in background job

Hello everyone While working on TUI for scripts, there there came the idea to' add a command' for dd too. That was, after 'wrapping' tar and wget either, to display their growing size and return the exit code with a textual-visual-feedback to the user. Now displaying the filesize of a... (13 Replies)
Discussion started by: sea
13 Replies

4. UNIX for Dummies Questions & Answers

302 server status code to 301/404 server status code

Hello, Sorry for my english. I have an arcade site. If database has the game name is good. if database has not the game name redirect to by 302 error code. if database has not the game name i want a 301/404 error code and no... (0 Replies)
Discussion started by: hoo
0 Replies

5. Shell Programming and Scripting

Extract URLs from HTML code using sed

Hello, i try to extract urls from google-search-results, but i have problem with sed filtering of html-code. what i wont is just list of urls thay apears between ........<p><a href=" and next following " in html code. here is my code, i use wget and pipelines to filtering. wget works, but... (13 Replies)
Discussion started by: L0rd
13 Replies

6. Solaris

Port status/Running daemon

Hi, I need to run an application (Hudson) listening to port 8080 on a remote Solaris server. I have managed to start that application and tried to access it with my browser from my local PC, but unsuccessfully. I need to find out what is blocking the access to that port (or any other). A... (9 Replies)
Discussion started by: JVerstry
9 Replies

7. Shell Programming and Scripting

How to know the status of process running in background

I have run one shell script in background that contains a endless while loop. I am not able to know the status of that job . Please provide any command to know this. I have already used "ps -aef" , "jobs" to know it , but it didn't work. I am sure the process is running as it is generating a file... (8 Replies)
Discussion started by: sumanta
8 Replies

8. UNIX for Dummies Questions & Answers

status of the script running under a particular owner?

How to track the status of a shell script running under a particular owner which is running in the background? (2 Replies)
Discussion started by: ishmael^soyuz
2 Replies

9. UNIX for Dummies Questions & Answers

Script to kill rsh processes running for more than 10 mins

Hi Friends, I need to write a script to kill some processes running for more than 10 minutes. Can I get some pointers on that. Thanks for ur help in Advance. Thanks&Regards, Amit (3 Replies)
Discussion started by: amitsayshii
3 Replies

10. Shell Programming and Scripting

Hw to Know the status of running JoB

Hi all, I am running a job .. and i want to know the status tht it is runnig or not .. and how can i find the jobId of my job .. I have to get it to kill my running job Pls let me know da Unix commands to do it .. i m wrking on Hp UNIX (1 Reply)
Discussion started by: ravi.sadani19
1 Replies
Login or Register to Ask a Question