Sponsored Content
Full Discussion: Downloading with Wget
Top Forums Shell Programming and Scripting Downloading with Wget Post 302523087 by keltonhalbert on Tuesday 17th of May 2011 11:23:54 PM
Old 05-18-2011
CPU & Memory Downloading with Wget

Hello everyone. I'm new both to the forum and to unix scripting, and this website has been very useful in putting together a script I am working on. However, I have run into a bit of a snag, which is why I have come here seeking help. First I will say what I am trying to do, and then what I have done so far.

I am trying to download weather model data for a meteorological program called GEMPAK, and the models update ever so often and filenames are based on timestamp and model type. An example filename is 2011051800_nam242.gem
These files take up a LOT of space (this directory alone is 21 gig!), so as you can imagine, after downloading the files I want to delete the old ones. Even more so, I do not need the *entire* directory, just the last 2 day's worth at most.

So, here is what I have done so far.
Code:
cd / ;
wget -S -N -l1 -r -np -A.gem http://metfs1.agron.iastate.edu/data/gempak/model/nam ;
find /metfs1.agron.iastate.edu/data/gempak/model/nam2 -Btime +4 -exec rm {} \;

Now I have the entire directory downloaded in the proper hierarchy, an the script will remove items older than 4 days. However, when the script runs again, it downloads the files I just deleted all over again, and I am trying not to keep too much space with files I don't need. So, my question is, can anyone help me find a way to either download 'x' amount of files, or only after 'x' timestamp? Or maybe another way I do not know about? Thank you very much in advance.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Downloading

Kindly help us... Is there a way in linux how we'll be able to check which workstation is downloading files from the internet and what file is it? Please put into consideration that our linux has a firewall. Thanks (1 Reply)
Discussion started by: eric_hing
1 Replies

2. Linux

Downloading Linux

My class "UNIX using Linux" started today. The software that came with my text book will not download. I am running XP pro .. When I load the "guide to UNIX using Linux" disk the only thing that comes up is E:/ showing some files that are on the disk and then it stops and does nothing. Can anyone... (13 Replies)
Discussion started by: Tammy
13 Replies

3. Linux

downloading linux

I can't download linux. why? :mad: (2 Replies)
Discussion started by: buddhika
2 Replies

4. Solaris

Downloading Solaris 10/08 ?

hi there, Has anybody else had any problems downloading Solaris 10/08 form the Sun web site ? I have tried a number of times, at different times of the day, and have the same results each time. Either the Sun Download manager times out after about 20 retries and I get nothing, or if I... (4 Replies)
Discussion started by: Freddo_21
4 Replies

5. Solaris

HTTP error while downloading solaris patches using wget

Hello, I am getting a HTTP error while downloading solaris patches using wget. 'Downloading unsigned patch 113096-03. --2010-06-18 03:51:15-- http://sunsolve.sun.com/pdownload.pl?target=113096-03&method=h Resolving sunsolve.sun.com (sunsolve.sun.com)... 192.18.108.40 Connecting to... (5 Replies)
Discussion started by: sunny_a_j
5 Replies

6. UNIX for Dummies Questions & Answers

Crontab Wget, downloading a file to a FTP

Hi there, I've got my own domain, ftp etc.. I'm using cPanel and I want to download a file periodically, every say 24 hours. I've used this command: wget -t inf http : / / www . somesite . com / webcam.jpg ftp : / / i @ MyDomain . net : Password @ ftp . MyDomain . net^no spaces... (24 Replies)
Discussion started by: zYx
24 Replies

7. UNIX for Dummies Questions & Answers

wget pdf downloading problem

Hi. I am trying to make a mirror of this free online journal: http://www.informaworld.com/smpp/title~content=t716100758~db=all Under the individual issues, the link location for the "Full Text PDF" does not have ".pdf" as an extension -- so when I use wget it misses the file. However clicking... (5 Replies)
Discussion started by: obo1234
5 Replies

8. Shell Programming and Scripting

Wget error while downloading from https website

Hi, I would like to download a file from a https website. I don't have the file name as it changes every day. I am using the following command: wget --no-check-certificate -r -np --user=ABC --password=DEF -O temp.txt https://<website/directory> I am getting followin error in my... (9 Replies)
Discussion started by: pinnacle
9 Replies

9. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403... (2 Replies)
Discussion started by: Prasannag87
2 Replies

10. Shell Programming and Scripting

Wget for downloading a public file (stream) as mp4

I need a hint for using wget for getting a free content from a TV station that is streaming its material for a while until it appears on any video platform, that means no use of illegal methods, because it is on air, recently published and available. But reading the manual for wget I tried the... (5 Replies)
Discussion started by: 1in10
5 Replies
4S-IMPORT(1J)							      4store							     4S-IMPORT(1J)

NAME
4s-import -- Import RDF into a 4store KB SYNOPSIS
4s-import kbname [-v] [-a] [-M default-model] [-m model] [-f format] [rdf-file] ... -v, --verbose Increase verbosity (by default success is silent) -a, --add Ordinarily importing RDF to a model replaces any previous contents of that model, by using the --add flag this behaviour is overridden and any triples which were previously in the model are retained. -M, --model-default Set a model (graph) URI which is used by default for all imported files -m, --model Set a model (graph) URI for the next named file only (overrides -M if it has been used) -f, --format Tell the RDF parser the format of the files (if not specified the parser will guess) SEE ALSO
4s-query(1), 4s-size(1), 4s-httpd(1), 4s-backend(1), 4s-delete-model(1) EXAMPLES
$ 4s-import demo one.ttl two.xrdf Imports two files into the demo KB, each as a separate model (graph) guessing the RDF format used for each file $ 4s-import demo -M http://example.com/menu starters.ttl mains.ttl sweets.ttl Imports three files into a single graph named <http://example.com/menu> 4store May 31, 2019 4store
All times are GMT -4. The time now is 08:54 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy