Downloading with Wget


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Downloading with Wget
# 1  
Old 05-18-2011
CPU & Memory Downloading with Wget

Hello everyone. I'm new both to the forum and to unix scripting, and this website has been very useful in putting together a script I am working on. However, I have run into a bit of a snag, which is why I have come here seeking help. First I will say what I am trying to do, and then what I have done so far.

I am trying to download weather model data for a meteorological program called GEMPAK, and the models update ever so often and filenames are based on timestamp and model type. An example filename is 2011051800_nam242.gem
These files take up a LOT of space (this directory alone is 21 gig!), so as you can imagine, after downloading the files I want to delete the old ones. Even more so, I do not need the *entire* directory, just the last 2 day's worth at most.

So, here is what I have done so far.
Code:
cd / ;
wget -S -N -l1 -r -np -A.gem http://metfs1.agron.iastate.edu/data/gempak/model/nam ;
find /metfs1.agron.iastate.edu/data/gempak/model/nam2 -Btime +4 -exec rm {} \;

Now I have the entire directory downloaded in the proper hierarchy, an the script will remove items older than 4 days. However, when the script runs again, it downloads the files I just deleted all over again, and I am trying not to keep too much space with files I don't need. So, my question is, can anyone help me find a way to either download 'x' amount of files, or only after 'x' timestamp? Or maybe another way I do not know about? Thank you very much in advance.
# 2  
Old 05-18-2011
I haven't tried this, but I think you could add -nc (or --no-clobber) to prevent it from downloading a file if one already exists, and then truncate rather than delete.

You'll end up with a bunch of zero-length files, but at least they don't use up disk space.

To truncate, this should work: -exec /bin/cp -f /dev/null {} \;

(I use /bin/cp instead of cp because I have cp aliased to "cp -i", and the "-i" overrides the "-f".)
This User Gave Thanks to KenJackson For This Post:
# 3  
Old 05-18-2011
Quote:
Originally Posted by KenJackson
I haven't tried this, but I think you could add -nc (or --no-clobber) to prevent it from downloading a file if one already exists, and then truncate rather than delete.

You'll end up with a bunch of zero-length files, but at least they don't use up disk space.

To truncate, this should work: -exec /bin/cp -f /dev/null {} \;

(I use /bin/cp instead of cp because I have cp aliased to "cp -i", and the "-i" overrides the "-f".)
Thanks for the response!
As far as -nc goes, I am not having trouble with it re-downloading the files while they exist, but after they are deleted. Unless of course I am misunderstanding you.

You'll have to pardon my noobishness, but what does -exec /bin/cp -f /dev/null {} \; do exactly? I'm trying learn as I go. I know -exec is for execute, but what is /bin/cp? or -f? or /dev/null? Thanks for your patience.

EDIT: So, I modified the code, and unfortunately wget is now comparing file sizes and re-downloads the data if the sizes do not match up with the directory. Any thoughts?
EDIT2: Got it all figured out. Now I know what -nc is for, and replaced -N with -nc. Thank you!

Last edited by keltonhalbert; 05-18-2011 at 01:32 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Wget for downloading a public file (stream) as mp4

I need a hint for using wget for getting a free content from a TV station that is streaming its material for a while until it appears on any video platform, that means no use of illegal methods, because it is on air, recently published and available. But reading the manual for wget I tried the... (5 Replies)
Discussion started by: 1in10
5 Replies

2. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403... (2 Replies)
Discussion started by: Prasannag87
2 Replies

3. Shell Programming and Scripting

Wget error while downloading from https website

Hi, I would like to download a file from a https website. I don't have the file name as it changes every day. I am using the following command: wget --no-check-certificate -r -np --user=ABC --password=DEF -O temp.txt https://<website/directory> I am getting followin error in my... (9 Replies)
Discussion started by: pinnacle
9 Replies

4. UNIX for Dummies Questions & Answers

wget pdf downloading problem

Hi. I am trying to make a mirror of this free online journal: http://www.informaworld.com/smpp/title~content=t716100758~db=all Under the individual issues, the link location for the "Full Text PDF" does not have ".pdf" as an extension -- so when I use wget it misses the file. However clicking... (5 Replies)
Discussion started by: obo1234
5 Replies

5. UNIX for Dummies Questions & Answers

Crontab Wget, downloading a file to a FTP

Hi there, I've got my own domain, ftp etc.. I'm using cPanel and I want to download a file periodically, every say 24 hours. I've used this command: wget -t inf http : / / www . somesite . com / webcam.jpg ftp : / / i @ MyDomain . net : Password @ ftp . MyDomain . net^no spaces... (24 Replies)
Discussion started by: zYx
24 Replies

6. Solaris

HTTP error while downloading solaris patches using wget

Hello, I am getting a HTTP error while downloading solaris patches using wget. 'Downloading unsigned patch 113096-03. --2010-06-18 03:51:15-- http://sunsolve.sun.com/pdownload.pl?target=113096-03&method=h Resolving sunsolve.sun.com (sunsolve.sun.com)... 192.18.108.40 Connecting to... (5 Replies)
Discussion started by: sunny_a_j
5 Replies

7. Solaris

Downloading Solaris 10/08 ?

hi there, Has anybody else had any problems downloading Solaris 10/08 form the Sun web site ? I have tried a number of times, at different times of the day, and have the same results each time. Either the Sun Download manager times out after about 20 retries and I get nothing, or if I... (4 Replies)
Discussion started by: Freddo_21
4 Replies

8. Linux

downloading linux

I can't download linux. why? :mad: (2 Replies)
Discussion started by: buddhika
2 Replies

9. Linux

Downloading Linux

My class "UNIX using Linux" started today. The software that came with my text book will not download. I am running XP pro .. When I load the "guide to UNIX using Linux" disk the only thing that comes up is E:/ showing some files that are on the disk and then it stops and does nothing. Can anyone... (13 Replies)
Discussion started by: Tammy
13 Replies

10. UNIX for Dummies Questions & Answers

Downloading

Kindly help us... Is there a way in linux how we'll be able to check which workstation is downloading files from the internet and what file is it? Please put into consideration that our linux has a firewall. Thanks (1 Reply)
Discussion started by: eric_hing
1 Replies
Login or Register to Ask a Question