Shell Programming and Scripting

I am new to shell scripting and need to write a program to copy files that are posted as links on a specific url. I want all the links copied with the same file name and the one posted on the webpage containing the url onto a specific directory. That is the first part. The second part of the script is to delete the files from the directory that are older than 3 days. Normally, I use "wget filename" from the specified directory then simply delete with the remove command manually. How would I set up the script to run automatically, twice daily? Thanks.

Do you know the name of the file you are downloading? Or just that the name will be on a web page?

If you know the exact name(s), you can just run the wget command like you've done before. (So what would be the question?) But if you have to get the name from a web page, you may have to scrape the name out of the html using grep, sed, or awk, etc.


For the second part, the simplest way is to do a loop with something like
sleep 12h
to delay. But then you couldn't log out.

A better way is to make a cron job, something like this:

Code:

echo '* 1,13 * * * $HOME/bin/myscript' | crontab

The first five parameters are minute, hour, day, month, day-of-week. 1,13 means run it at 1:00 AM and 1:00 PM.

Don't put any loop in myscript, just run once and exit. Cron will run it again in 12 hours.

There are numerous files that must be downloaded from the site twice daily. The file names change based on date. Looking at the system, I am thinking the the script must also remove the files first before downloading new ones because there will be some overlap in the dates. The site I am downloading the files from is Index of /tiziana/rhuff there are two subdirectories which will be based on the year month and date. The latter contains the more recent files. I wanted to store the files for three days but I don't think that is possible unless I rename the files in the script prior to downloading the new ones. So just to keep it simple lets say I remove all files prior to copying the new ones. So there are three things that need to be done.

1.Remove the current files.
That could be done with a simple remove all files from directory (but keep the directory) command which I am not sure how to do.

2. Have a command copy ALL FILES and keep the same file name conventions stored on a specified directory.

3. Repeat twice daily at a specified time.

I understand what you are saying about the crontab function. Am I correct in assuming that I would need to write a script that does functions 1 & 2 and then enter the function manually that used the crontab function to initiate the script automatically with the name of the script being bin/myscript based on your prior post? If so, How would I write the code for the first two steps?