Visit Our UNIX and Linux User Community


Using wget and check for non-zero files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using wget and check for non-zero files
# 1  
Old 10-18-2007
Using wget and check for non-zero files

Hello,

I'm using a shell script containing a wget-command that copies html-files from a website to my ISP-server. I therefore want to check if that file exist and also if the filessize is larger than e.g. 100 bytes. Probably it's done by using something like fileexist and filesize.

The wget -commandline looks like:
Code:
wget http://www.test.nl/test.html -O /home/myname/domains/myname.nl/public_html/data/test2.html --quiet

But I have a beginners experience in shell-scripting so can you help me out?!
# 2  
Old 10-18-2007
eg: wget --spider -v http://images.ucomics.com/comics/ga/2007/ga071017.gif"

$man wget
When invoked with --spider option,Wget will behave as a Web Spider,which means that it will not download the pages, just check that
they are there.

-v is for verbose output, as always
# 3  
Old 10-18-2007
Thanks for the quick reply!

So the script will now look like:

Code:
wget --spider -v http://www.test.nl/test.html
wget http://www.test.nl/test.html -O /home/myname/domains/myname.nl/public_html/data/test2.html --quiet

But still gets files if they are zero bytes and overwrites existing files. That's not what I want. I want to check the existence (using your spider option) and I want to allow overwriting in case the files to be copied are not empty.

Can you help me out?!
# 4  
Old 08-12-2008
Quote:
Originally Posted by weatherboys
Thanks for the quick reply!

So the script will now look like:

Code:
wget --spider -v http://www.test.nl/test.html
wget http://www.test.nl/test.html -O /home/myname/domains/myname.nl/public_html/data/test2.html --quiet

But still gets files if they are zero bytes and overwrites existing files. That's not what I want. I want to check the existence (using your spider option) and I want to allow overwriting in case the files to be copied are not empty.

Can you help me out?!
can you post the output of wget --spider -v http://www.test.nl/test.html ? Perhaps you just need to redirect that out put to a txt file run a few awk lines then check the file with if for the information you want and then download the file.. or not download it.

edit: sorry for posting this.. didnt realize it was a year old thread. Not sure how I even came across it Smilie

Previous Thread | Next Thread
Test Your Knowledge in Computers #162
Difficulty: Easy
Early versions of MINIX were created by Andrew S. Tanenbaum for commercial purposes.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

CMD to check status of the server using Wget

Hi All, Using Wget I'm able to get the status of the server.....only when the server is completely down or up.... but problem here in script is Suppose if the server got hang I mean to say that if the server is taking long time to login, for example normally the server takes 3 seconds to login... (3 Replies)
Discussion started by: manohar2013
3 Replies

2. Shell Programming and Scripting

Check wget return code

hello check this script it jump to else part, in n both cases, (if files exist or not) wget $MIRROR/kkk.zip && wget $MIRROR/jjj.zip RC="$?" if ] then echo -e "$RED Ooops, Download links are broken...! $RESET" else echo -e "$GREEN Everything is fine, Cheers ... $RESET" fi (4 Replies)
Discussion started by: nimafire
4 Replies

3. Shell Programming and Scripting

Wget - how to ignore files in immediate directory?

i am trying to recursively save a remote FTP server but exclude the files immediately under a directory directory1 wget -r -N ftp://user:pass@hostname/directory1 I want to keep these which may have more files under them directory1/dir1/file.jpg directory1/dir2/file.jpg... (16 Replies)
Discussion started by: vanessafan99
16 Replies

4. Shell Programming and Scripting

Files download using wget

Hi, I need to implement below logic to download files daily from a URL. * Need to check if it is yesterday's file (YYYY-DD-MM.dat) * If present then download from URL (sample_url/2013-01-28.dat) * Need to implement wait logic if not present * if it still not able to find the file... (1 Reply)
Discussion started by: rakesh5300
1 Replies

5. Shell Programming and Scripting

Check if wget finishes

how can you check if wget finishes? i have this code to store the source of a website url into a variable: source=$(wget --timeout=15 -qO - $site) but this gets stuck sometimes e.g., one site had a virus and it stopped the script how can i check if wget finishes? i tried to run... (5 Replies)
Discussion started by: vanessafan99
5 Replies

6. UNIX for Dummies Questions & Answers

Problem with wget no check certificate.

Hi, I'm trying to install some libraries, when running the makefile I get an error from the "wget --no check certificate option". I had a look help and the option wasn't listed. Anyone know what I'm missing. (0 Replies)
Discussion started by: davcra
0 Replies

7. Shell Programming and Scripting

Help with WGET and renaming downloaded files :(

Hi everybody, I would greatly appreciate some expertise in this matter. I am trying find an efficient way to batch download files from a website and rename each file with the url it originated from (from the CLI). (ie. Instead of xyz.zip, the output file would be http://www.abc.com/xyz.zip) A... (10 Replies)
Discussion started by: o0110o
10 Replies

8. Shell Programming and Scripting

how to limit files downloaded by wget

I am trying to download a page and retrieve only wav and mp3 files via wget. the website is: Alarm Sounds | Free Sound Effects | Alarm Sound Clips | Sound Bites my command is : wget -rl 2 -e robots=off -A wav,mp3 http://soundbible.com/tags-alarm.html When not using the -A wav,mp3... (2 Replies)
Discussion started by: Narnie
2 Replies

9. Shell Programming and Scripting

wget skips certain files.

I am trying to use wget to automate downloading of some mp3/wav files. However, I can't get it to follow the link to the mp3s. This is the line (it is not really the website): wget -prl 1 http://website.com/alarmsHowever, if I right-click and copy the link on the webpage in firefox, then... (4 Replies)
Discussion started by: Narnie
4 Replies

10. Shell Programming and Scripting

wget to check an URL

I all, I wrote an script which starts a Weblogic server and waits until its loaded to deploy several apps. The way I checked was something like: while ; do wget --spider <URL>:<port>/console > /dev/null 2>&1 rc=$? done This works perfectly because it's an HTML site and when server is... (2 Replies)
Discussion started by: AlbertGM
2 Replies

Featured Tech Videos