Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Script to if webpage is: forbidden, Error...

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Script to if webpage is: forbidden, Error...
# 1  
Old 05-03-2013
Java Script to if webpage is: forbidden, Error...
I have a file with about 29,000 website names. I would like to find a script that would test if each website name is current/active or not. If not it should check if the page is: Forbidden, 404 Error...
# 2  
Old 05-03-2013
You can have something like this.

A url file named file_urls

hitendra@nabla:~$ cat file_urls
http://unixchamp.com/allowed
http://unixchamp.com/forbidden
http://unixchamp.com/doesnotexists
hitendra@nabla:~$


and a script test_webpage.sh

hitendra@nabla:~$ cat test_webpage.sh
#/bin/bash
for url in `cat file_urls`
do
responses=`wget --server-response $url 2>&1|grep HTTP/|awk '{ print $2 }'`
echo "$responses : $url"
done
hitendra@nabla:~$


output will be the status code for each of the url

hitendra@nabla:~$ ./test_webpage.sh
301
200 : http://unixchamp.com/allowed
403 : http://unixchamp.com/forbidden
404 : http://unixchamp.com/doesnotexists
hitendra@nabla:~$


based on the http response code , you can decide about the status of website.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Wget-403-Forbidden-Error

Hi Friends, I did an extensive search over the internet and tried all possible solutions that were recommended, but couldn't figure this out. Please see this link http://www.dli.gov.in/data6/upload/0159/808/PTIFF/00000007.tif It works. But, when I try the following command wget -r... (7 Replies)
Discussion started by: jacobs.smith
7 Replies

2. Web Development

Directory index forbidden by Options directive error on specific directory with indexing disabled

I am seeing the following error appear numerous times in my Apache error log: I have my Apache config configured as below, so I would expect indexing not to occur on this directory as it falls under the parent /web directory. Strangely all the IP address, including this example, all... (5 Replies)
Discussion started by: crmpicco
5 Replies

3. Web Development

forbidden error when accessing file(s) on a unix filesystem

greetings, i was recently asked to install apache1.3.31 on an aix5.3 server so users could have access to their output files. these two directories are nfs filesystems served by a netapp filer mounted on the aix machine. i've made no changes to the apache config and it's running as it came out of... (0 Replies)
Discussion started by: crimso
0 Replies

4. Shell Programming and Scripting

Use Webpage to Start Script

I apologize if this is in the incorrect section, I'm not quite sure which section it should go in. Anyways... I've got a script that I'd like to be able to start with a webpage, something that just has a button that says "Start this Bot", which will start the bot and put it in the background.... (4 Replies)
Discussion started by: JoeGazz84
4 Replies

5. Shell Programming and Scripting

open a webpage through perl script

Hi All Can anyone please help me how to open a webpage suppose(Google) with the help of perl script and refresh it after every 5 minutes. (0 Replies)
Discussion started by: parthmittal2007
0 Replies

6. Shell Programming and Scripting

Read webpage from shell script

Hey experts, I am trying to read a webpage and want to search for a text patter (case insensitive). Please help me acquire this. If the text is found i will receive a mail which can be hardcoded in the script. I am not a big fan of using PERL script. Please help. Machine: AIX... (15 Replies)
Discussion started by: bankimmehta
15 Replies

7. Shell Programming and Scripting

Wget-Forbidden error

Hi , I am using shell script to connect to a https site and download all the files in a directory. eg: wget --no-check-certificate -r -np --user=<username> --password=<passwd> -O temp.txt https:<website>/<dirname> Error : HTTP request sent, awaiting response... 403 Forbidden 07:40:20... (0 Replies)
Discussion started by: mohanpadamata
0 Replies

8. Shell Programming and Scripting

Run a script from a webpage

Hello, I was trying to run a shell script when I click on webpage link. My webpage is hosted on Sun One web server. This is something I was looking, Chris Johnson's Test Bed I already have my shell script and it works fine when I run it from the command line. Can somebody tell me what are... (3 Replies)
Discussion started by: grajp002
3 Replies

9. Shell Programming and Scripting

Help with perl script to search webpage

i have to point out that i'm a avid shell script lover and i have next to zero interest in perl which is why i dont know the first thing about it. however, just need to do this one thing on it.. i need to do something in perl but dont know how. I have a list of email addresses in the following... (5 Replies)
Discussion started by: Terrible
5 Replies

10. Linux

Apache Forbidden Error

When i go to my browser and type both localhost or the ip, i get Forbidden, you do not have permission to access / on this machine Whys that ? (1 Reply)
Discussion started by: perleo
1 Replies
Login or Register to Ask a Question