Downloading with Wget Post: 302523087

Sponsored Content

Top Forums Shell Programming and Scripting Downloading with Wget Post 302523087 by keltonhalbert on Tuesday 17th of May 2011 11:23:54 PM

05-18-2011

Registered User

Downloading with Wget

Hello everyone. I'm new both to the forum and to unix scripting, and this website has been very useful in putting together a script I am working on. However, I have run into a bit of a snag, which is why I have come here seeking help. First I will say what I am trying to do, and then what I have done so far.

I am trying to download weather model data for a meteorological program called GEMPAK, and the models update ever so often and filenames are based on timestamp and model type. An example filename is 2011051800_nam242.gem
These files take up a LOT of space (this directory alone is 21 gig!), so as you can imagine, after downloading the files I want to delete the old ones. Even more so, I do not need the *entire* directory, just the last 2 day's worth at most.

So, here is what I have done so far.

Code:

cd / ;
wget -S -N -l1 -r -np -A.gem http://metfs1.agron.iastate.edu/data/gempak/model/nam ;
find /metfs1.agron.iastate.edu/data/gempak/model/nam2 -Btime +4 -exec rm {} \;

Now I have the entire directory downloaded in the proper hierarchy, an the script will remove items older than 4 days. However, when the script runs again, it downloads the files I just deleted all over again, and I am trying not to keep too much space with files I don't need. So, my question is, can anyone help me find a way to either download 'x' amount of files, or only after 'x' timestamp? Or maybe another way I do not know about? Thank you very much in advance.

keltonhalbert

View Public Profile for keltonhalbert

Find all posts by keltonhalbert

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Downloading

Kindly help us... Is there a way in linux how we'll be able to check which workstation is downloading files from the internet and what file is it? Please put into consideration that our linux has a firewall. Thanks

2. Linux

Downloading Linux

My class "UNIX using Linux" started today. The software that came with my text book will not download. I am running XP pro .. When I load the "guide to UNIX using Linux" disk the only thing that comes up is E:/ showing some files that are on the disk and then it stops and does nothing. Can anyone...

3. Linux

downloading linux

I can't download linux. why? :mad:

4. Solaris

Downloading Solaris 10/08 ?

hi there, Has anybody else had any problems downloading Solaris 10/08 form the Sun web site ? I have tried a number of times, at different times of the day, and have the same results each time. Either the Sun Download manager times out after about 20 retries and I get nothing, or if I...

5. Solaris

HTTP error while downloading solaris patches using wget

Hello, I am getting a HTTP error while downloading solaris patches using wget. 'Downloading unsigned patch 113096-03. --2010-06-18 03:51:15-- http://sunsolve.sun.com/pdownload.pl?target=113096-03&method=h Resolving sunsolve.sun.com (sunsolve.sun.com)... 192.18.108.40 Connecting to...

6. UNIX for Dummies Questions & Answers

Crontab Wget, downloading a file to a FTP

Hi there, I've got my own domain, ftp etc.. I'm using cPanel and I want to download a file periodically, every say 24 hours. I've used this command: wget -t inf http : / / www . somesite . com / webcam.jpg ftp : / / i @ MyDomain . net : Password @ ftp . MyDomain . net^no spaces...

7. UNIX for Dummies Questions & Answers

wget pdf downloading problem

Hi. I am trying to make a mirror of this free online journal: http://www.informaworld.com/smpp/title~content=t716100758~db=all Under the individual issues, the link location for the "Full Text PDF" does not have ".pdf" as an extension -- so when I use wget it misses the file. However clicking...

8. Shell Programming and Scripting

Wget error while downloading from https website

Hi, I would like to download a file from a https website. I don't have the file name as it changes every day. I am using the following command: wget --no-check-certificate -r -np --user=ABC --password=DEF -O temp.txt https://<website/directory> I am getting followin error in my...

9. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403...

10. Shell Programming and Scripting

Wget for downloading a public file (stream) as mp4

I need a hint for using wget for getting a free content from a TV station that is streaming its material for a while until it appears on any video platform, that means no use of illegal methods, because it is on air, recently published and available. But reading the manual for wget I tried the...

LEARN ABOUT LINUX

service

service(8)						      System Manager's Manual							service(8)

NAME

       service - run a System V init script

SYNOPSIS

       service SCRIPT COMMAND [OPTIONS]

       service --status-all

       service --help | -h | --version

DESCRIPTION

       service runs a System V init script in as predictable environment as possible, removing most environment variables and with current working
       directory set to /.

       The SCRIPT parameter specifies a System V init script, located in /etc/init.d/SCRIPT.  The  supported  values  of  COMMAND  depend  on  the
       invoked	script,  service  passes  COMMAND and OPTIONS it to the init script unmodified.  All scripts should support at least the start and
       stop commands.  As a special case, if COMMAND is --full-restart, the script is run twice, first with the stop command, then with the  start
       command.

       service --status-all runs all init scripts, in alphabetical order, with the status command.

EXIT CODES

       service calls the init script and returns the status returned by it.

FILES

       /etc/init.d
	      The directory containing System V init scripts.

ENVIRONMENT

       LANG, TERM
	      The only environment variables passed to the init scripts.

SEE ALSO

       /etc/init.d/skeleton,
       update-rc.d(8),
       init(8),
       invoke-rc.d(8).

								     Jan 2006								service(8)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Downloading

Discussion started by: eric_hing

2. Linux

Downloading Linux

Discussion started by: Tammy

3. Linux

downloading linux

Discussion started by: buddhika

4. Solaris

Downloading Solaris 10/08 ?

Discussion started by: Freddo_21