Sponsored Content
Full Discussion: Downloading with Wget
Top Forums Shell Programming and Scripting Downloading with Wget Post 302523087 by keltonhalbert on Tuesday 17th of May 2011 11:23:54 PM
Old 05-18-2011
CPU & Memory Downloading with Wget

Hello everyone. I'm new both to the forum and to unix scripting, and this website has been very useful in putting together a script I am working on. However, I have run into a bit of a snag, which is why I have come here seeking help. First I will say what I am trying to do, and then what I have done so far.

I am trying to download weather model data for a meteorological program called GEMPAK, and the models update ever so often and filenames are based on timestamp and model type. An example filename is 2011051800_nam242.gem
These files take up a LOT of space (this directory alone is 21 gig!), so as you can imagine, after downloading the files I want to delete the old ones. Even more so, I do not need the *entire* directory, just the last 2 day's worth at most.

So, here is what I have done so far.
Code:
cd / ;
wget -S -N -l1 -r -np -A.gem http://metfs1.agron.iastate.edu/data/gempak/model/nam ;
find /metfs1.agron.iastate.edu/data/gempak/model/nam2 -Btime +4 -exec rm {} \;

Now I have the entire directory downloaded in the proper hierarchy, an the script will remove items older than 4 days. However, when the script runs again, it downloads the files I just deleted all over again, and I am trying not to keep too much space with files I don't need. So, my question is, can anyone help me find a way to either download 'x' amount of files, or only after 'x' timestamp? Or maybe another way I do not know about? Thank you very much in advance.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Downloading

Kindly help us... Is there a way in linux how we'll be able to check which workstation is downloading files from the internet and what file is it? Please put into consideration that our linux has a firewall. Thanks (1 Reply)
Discussion started by: eric_hing
1 Replies

2. Linux

Downloading Linux

My class "UNIX using Linux" started today. The software that came with my text book will not download. I am running XP pro .. When I load the "guide to UNIX using Linux" disk the only thing that comes up is E:/ showing some files that are on the disk and then it stops and does nothing. Can anyone... (13 Replies)
Discussion started by: Tammy
13 Replies

3. Linux

downloading linux

I can't download linux. why? :mad: (2 Replies)
Discussion started by: buddhika
2 Replies

4. Solaris

Downloading Solaris 10/08 ?

hi there, Has anybody else had any problems downloading Solaris 10/08 form the Sun web site ? I have tried a number of times, at different times of the day, and have the same results each time. Either the Sun Download manager times out after about 20 retries and I get nothing, or if I... (4 Replies)
Discussion started by: Freddo_21
4 Replies

5. Solaris

HTTP error while downloading solaris patches using wget

Hello, I am getting a HTTP error while downloading solaris patches using wget. 'Downloading unsigned patch 113096-03. --2010-06-18 03:51:15-- http://sunsolve.sun.com/pdownload.pl?target=113096-03&method=h Resolving sunsolve.sun.com (sunsolve.sun.com)... 192.18.108.40 Connecting to... (5 Replies)
Discussion started by: sunny_a_j
5 Replies

6. UNIX for Dummies Questions & Answers

Crontab Wget, downloading a file to a FTP

Hi there, I've got my own domain, ftp etc.. I'm using cPanel and I want to download a file periodically, every say 24 hours. I've used this command: wget -t inf http : / / www . somesite . com / webcam.jpg ftp : / / i @ MyDomain . net : Password @ ftp . MyDomain . net^no spaces... (24 Replies)
Discussion started by: zYx
24 Replies

7. UNIX for Dummies Questions & Answers

wget pdf downloading problem

Hi. I am trying to make a mirror of this free online journal: http://www.informaworld.com/smpp/title~content=t716100758~db=all Under the individual issues, the link location for the "Full Text PDF" does not have ".pdf" as an extension -- so when I use wget it misses the file. However clicking... (5 Replies)
Discussion started by: obo1234
5 Replies

8. Shell Programming and Scripting

Wget error while downloading from https website

Hi, I would like to download a file from a https website. I don't have the file name as it changes every day. I am using the following command: wget --no-check-certificate -r -np --user=ABC --password=DEF -O temp.txt https://<website/directory> I am getting followin error in my... (9 Replies)
Discussion started by: pinnacle
9 Replies

9. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403... (2 Replies)
Discussion started by: Prasannag87
2 Replies

10. Shell Programming and Scripting

Wget for downloading a public file (stream) as mp4

I need a hint for using wget for getting a free content from a TV station that is streaming its material for a while until it appears on any video platform, that means no use of illegal methods, because it is on air, recently published and available. But reading the manual for wget I tried the... (5 Replies)
Discussion started by: 1in10
5 Replies
IFP(1)							      General Commands Manual							    IFP(1)

NAME
ifp - access iRiver iFP audio devices SYNOPSIS
ifp command [arguments ...] DESCRIPTION
ifp lets you manage your music on an iRiver iFP music player acting in "Manager Mode". If your player is using "UMS Mode", then you don't need this program; it will appear as new drive when you plug it in. This manual page documents the version of ifp included as an example application for the libifp library. The interface is intended to mimic the ifp binary found in the ifp-line binary package. ifp can upload or download files or directories, delete or make directories on the device, format the device, or upgrade your firmware. COMMANDS
ls [directory] List the files and directories on the device. The root directory is /. Directories are preceded with 'd' and other files with 'f'. df Display total and free space on the device. upload localfile remotefile Copy the file named localfile to the iFP device, and name it remotefile. (This is like cp.) upload localfile remotedir Copy the file named localfile to the iFP device, into remotedir. (This is like cp.) upload localdir remotedir Copy the entire directory localdir to the device, into remotedir. This copies into a subdirectory, so upload foo / puts files from the directory foo info /foo on the device. (This is like cp -R.) This uploads the entire directory, not just audio files. put localfile | localdir Upload the file or directory to the device, with the same name it has locally. If a directory is given, it is put in /. download remotefile localfile download remotefile localdir download remotedir localdir Like upload, but copies from the iFP device to your system. Some proprietary file types may not be downloaded. get remotefile | remotedir Like put, but copies files or directories from the device to your current working directory. rm [-r] file Delete (recursively) a file (or directory) on the device. rmdir dir Delete an empty directory on the device. mkdir dir Create a directory on the device. battery Display battery status of the device. typestring Display the model number of the device. firmversion Display the firmware revision currently on the device. format Reformat the device's memory. This will delete all your music. firmupdate FIRMWARE.HEX Upload the file FIRMWARE.HEX as new firmware for the device. This will not delete your music, but may have many other detrimental effects. Don't turn the device off or unplug it while the firmware is updating, and don't attempt to upload new firmware with a low battery. AUTHORS
libifp was written by Geoff Oakham, based on ifp by Yamashiro Jun. This manual page was written for Debian by Joe Wreschnig <piman@debian.org>, but may be used by others. May 30th, 2004 IFP(1)
All times are GMT -4. The time now is 09:54 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy