Hello everyone. I'm new both to the forum and to unix scripting, and this website has been very useful in putting together a script I am working on. However, I have run into a bit of a snag, which is why I have come here seeking help. First I will say what I am trying to do, and then what I have done so far.
I am trying to download weather model data for a meteorological program called GEMPAK, and the models update ever so often and filenames are based on timestamp and model type. An example filename is 2011051800_nam242.gem
These files take up a LOT of space (this directory alone is 21 gig!), so as you can imagine, after downloading the files I want to delete the old ones. Even more so, I do not need the *entire* directory, just the last 2 day's worth at most.
So, here is what I have done so far.
Now I have the entire directory downloaded in the proper hierarchy, an the script will remove items older than 4 days. However, when the script runs again, it downloads the files I just deleted all over again, and I am trying not to keep too much space with files I don't need. So, my question is, can anyone help me find a way to either download 'x' amount of files, or only after 'x' timestamp? Or maybe another way I do not know about? Thank you very much in advance.
I haven't tried this, but I think you could add -nc (or --no-clobber) to prevent it from downloading a file if one already exists, and then truncate rather than delete.
You'll end up with a bunch of zero-length files, but at least they don't use up disk space.
To truncate, this should work: -exec /bin/cp -f /dev/null {} \;
(I use /bin/cp instead of cp because I have cp aliased to "cp -i", and the "-i" overrides the "-f".)
This User Gave Thanks to KenJackson For This Post:
I haven't tried this, but I think you could add -nc (or --no-clobber) to prevent it from downloading a file if one already exists, and then truncate rather than delete.
You'll end up with a bunch of zero-length files, but at least they don't use up disk space.
To truncate, this should work: -exec /bin/cp -f /dev/null {} \;
(I use /bin/cp instead of cp because I have cp aliased to "cp -i", and the "-i" overrides the "-f".)
Thanks for the response!
As far as -nc goes, I am not having trouble with it re-downloading the files while they exist, but after they are deleted. Unless of course I am misunderstanding you.
You'll have to pardon my noobishness, but what does -exec /bin/cp -f /dev/null {} \; do exactly? I'm trying learn as I go. I know -exec is for execute, but what is /bin/cp? or -f? or /dev/null? Thanks for your patience.
EDIT: So, I modified the code, and unfortunately wget is now comparing file sizes and re-downloads the data if the sizes do not match up with the directory. Any thoughts?
EDIT2: Got it all figured out. Now I know what -nc is for, and replaced -N with -nc. Thank you!
Last edited by keltonhalbert; 05-18-2011 at 01:32 AM..
I need a hint for using wget for getting a free content from a TV station that is streaming its material for a while until it appears on any video platform, that means no use of illegal methods, because it is on air, recently published and available. But reading the manual for wget I tried the... (5 Replies)
Hi,
I need to download a zip file from my the below US govt link.
https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP
I only have wget utility installed on the server.
When I use the below command, I am getting error 403... (2 Replies)
Hi,
I would like to download a file from a https website. I don't have the file name as it changes every day.
I am using the following command:
wget --no-check-certificate -r -np --user=ABC --password=DEF -O temp.txt https://<website/directory>
I am getting followin error in my... (9 Replies)
Hi. I am trying to make a mirror of this free online journal:
http://www.informaworld.com/smpp/title~content=t716100758~db=all
Under the individual issues, the link location for the "Full Text PDF" does not have ".pdf" as an extension -- so when I use wget it misses the file. However clicking... (5 Replies)
Hi there,
I've got my own domain, ftp etc.. I'm using cPanel and I want to download a file periodically, every say 24 hours.
I've used this command:
wget -t inf http : / / www . somesite . com / webcam.jpg
ftp : / / i @ MyDomain . net : Password @ ftp . MyDomain . net^no spaces... (24 Replies)
Hello,
I am getting a HTTP error while downloading solaris patches using wget.
'Downloading unsigned patch 113096-03.
--2010-06-18 03:51:15-- http://sunsolve.sun.com/pdownload.pl?target=113096-03&method=h
Resolving sunsolve.sun.com (sunsolve.sun.com)... 192.18.108.40
Connecting to... (5 Replies)
hi there,
Has anybody else had any problems downloading Solaris 10/08 form the Sun web site ?
I have tried a number of times, at different times of the day, and have the same results each time.
Either the Sun Download manager times out after about 20 retries and I get nothing, or if I... (4 Replies)
My class "UNIX using Linux" started today. The software that came with my text book will not download. I am running XP pro .. When I load the "guide to UNIX using Linux" disk the only thing that comes up is E:/ showing some files that are on the disk and then it stops and does nothing. Can anyone... (13 Replies)
Kindly help us... Is there a way in linux how we'll be able to check which workstation is downloading files from the internet and what file is it? Please put into consideration that our linux has a firewall.
Thanks (1 Reply)