shell script to download files from a site?


 
Thread Tools Search this Thread
Operating Systems Linux shell script to download files from a site?
# 1  
Old 11-26-2008
shell script to download files from a site?

Hey everyone, my wife has purchased a bundle package of a bunch of images from a site, and now has to download each one of them manually. There are about 500 downloads, and it's quite a hassle to browse to each page and download them all individually.

I would like to write a shell script to just go out and grab all the links and do it all automatically, but one part of the script eludes me.

She has an email that contains the initial links to all the download pages. I'm pretty sure I could manipulate the data in that page, and get all those links out and into a file. Then from there, I would do a for loop and loop through that file. So what I would need to do from there is have the script go to each link, then get the source code from that page. At that point, I could manipulate the data in that file and grab the link for the file in there. This is where it gets a little tricky. I can't just tell it to download that link, because it's just a php page that initiates the download for you. So I would have to have it hit the link, and tell it to save the file somewhere.

If all I could do was get it to hit the link for me, that would still save a lot of time. Then we could just sit there while it runs, and click "Save" on every one.

Anyone have any ideas?

Thanks.
# 2  
Old 11-26-2008
If you have firefox you can use the add-on DownThemAll. Excellent for this job!

Regards
# 3  
Old 11-26-2008
Cool addon.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Download a site

A buddy of mine was telling me last night that you can write a bash script that will download an entire site in gedit?? Is this true??? I think I am going to fall in love with bash :D Any good tutorials?? (15 Replies)
Discussion started by: graphicsman
15 Replies

2. Shell Programming and Scripting

lftp script to connect to external sftp site and download to internal ftp and then send email

Hi there, I'm new to shell scripting and need some help if possible? I need to create a shell script (.sh) to run as a cron job on an ubuntu linux server to connect to an external sftp sites directory using credentials (which I have) and then download to our internal ftp server and then copy... (3 Replies)
Discussion started by: ghath
3 Replies

3. Shell Programming and Scripting

Shell Script for Upload/download files using cURL

hi please help me out here, i want to use curl command in shell script to test web pages, what i have is an opening page, when i click on a button on opening page, the next page comes up and then i have to upload a file n then click another button to submit and then comes the output page,... (2 Replies)
Discussion started by: Olivia
2 Replies

4. Shell Programming and Scripting

Wget, download file from site's folder.

Ok, this is quite weird. wget -r mysite.com/mylink/ should get all the files recursively from the 'mylink' folder. The problem is that wget saves an index.html file! When I open this index.html with my browser I realize that it shows all the files in the current folder (plus an option to move... (3 Replies)
Discussion started by: hakermania
3 Replies

5. Shell Programming and Scripting

Shell script to automatically download files

I am new to shell scripting and need to write a program to copy files that are posted as links on a specific url. I want all the links copied with the same file name and the one posted on the webpage containing the url onto a specific directory. That is the first part. The second part of the script... (2 Replies)
Discussion started by: libertyforall
2 Replies

6. UNIX for Dummies Questions & Answers

Download image every 24hrs + put it in php site

Hi everyone, I am all new to this and I might not know what I am asking about :) I have access to a cPanel with cron tab scheduler. I know nothing re scripting creating php etc.. I own a ftp mainly for own email address, some storage, and a forum (ready-made one :P ) I have some knowledge... (5 Replies)
Discussion started by: zYx
5 Replies

7. Shell Programming and Scripting

Reconnect and download data from online site

Hi, I have to connect to online site and download some data from the online site but sometimes the site which i have to connect will be busy or it will not be working in such a case i have to connect atleast 3 times and try to download the data. I tried like this: use CGI::Carp... (2 Replies)
Discussion started by: vanitham
2 Replies

8. Shell Programming and Scripting

shell script to download variables files

Hello all i am working on creating shell script to download files daily example : file12_10_2009.txt.gz next day this file will be file13_10_2009.txt.gz and so on.. i need help to know how to download this incrimental date files daily ? regards (1 Reply)
Discussion started by: mogabr
1 Replies

9. Shell Programming and Scripting

script for download files from ftp site

I'm new to scripting. I'm trying to write a script to download files from ftp site, the following is the script and the message i get after running the script. no files were downloaded :( Thanks advance! script: #!/usr/bin/ksh DAY=`date --date="-1 days" +%y%m%d` ftp -v -n "ftp.address" <<... (5 Replies)
Discussion started by: tiff-matt
5 Replies

10. Shell Programming and Scripting

Pulling a list of files from FTP site in a shell script

Hi, I am writting a shell script which will pull a list files (mentioned in one file 1.txt) from external FTP site (for ex: ftp://abcd.efghijk.com/). The 1.txt is in my local unix directory. I have username and password to connect the external FTP site. Also before I pull the files, I need... (0 Replies)
Discussion started by: spatra
0 Replies
Login or Register to Ask a Question