Sponsored Content
Operating Systems Linux shell script to download files from a site? Post 302262207 by paqman on Wednesday 26th of November 2008 03:28:56 PM
Old 11-26-2008
shell script to download files from a site?

Hey everyone, my wife has purchased a bundle package of a bunch of images from a site, and now has to download each one of them manually. There are about 500 downloads, and it's quite a hassle to browse to each page and download them all individually.

I would like to write a shell script to just go out and grab all the links and do it all automatically, but one part of the script eludes me.

She has an email that contains the initial links to all the download pages. I'm pretty sure I could manipulate the data in that page, and get all those links out and into a file. Then from there, I would do a for loop and loop through that file. So what I would need to do from there is have the script go to each link, then get the source code from that page. At that point, I could manipulate the data in that file and grab the link for the file in there. This is where it gets a little tricky. I can't just tell it to download that link, because it's just a php page that initiates the download for you. So I would have to have it hit the link, and tell it to save the file somewhere.

If all I could do was get it to hit the link for me, that would still save a lot of time. Then we could just sit there while it runs, and click "Save" on every one.

Anyone have any ideas?

Thanks.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Pulling a list of files from FTP site in a shell script

Hi, I am writting a shell script which will pull a list files (mentioned in one file 1.txt) from external FTP site (for ex: ftp://abcd.efghijk.com/). The 1.txt is in my local unix directory. I have username and password to connect the external FTP site. Also before I pull the files, I need... (0 Replies)
Discussion started by: spatra
0 Replies

2. Shell Programming and Scripting

script for download files from ftp site

I'm new to scripting. I'm trying to write a script to download files from ftp site, the following is the script and the message i get after running the script. no files were downloaded :( Thanks advance! script: #!/usr/bin/ksh DAY=`date --date="-1 days" +%y%m%d` ftp -v -n "ftp.address" <<... (5 Replies)
Discussion started by: tiff-matt
5 Replies

3. Shell Programming and Scripting

shell script to download variables files

Hello all i am working on creating shell script to download files daily example : file12_10_2009.txt.gz next day this file will be file13_10_2009.txt.gz and so on.. i need help to know how to download this incrimental date files daily ? regards (1 Reply)
Discussion started by: mogabr
1 Replies

4. Shell Programming and Scripting

Reconnect and download data from online site

Hi, I have to connect to online site and download some data from the online site but sometimes the site which i have to connect will be busy or it will not be working in such a case i have to connect atleast 3 times and try to download the data. I tried like this: use CGI::Carp... (2 Replies)
Discussion started by: vanitham
2 Replies

5. UNIX for Dummies Questions & Answers

Download image every 24hrs + put it in php site

Hi everyone, I am all new to this and I might not know what I am asking about :) I have access to a cPanel with cron tab scheduler. I know nothing re scripting creating php etc.. I own a ftp mainly for own email address, some storage, and a forum (ready-made one :P ) I have some knowledge... (5 Replies)
Discussion started by: zYx
5 Replies

6. Shell Programming and Scripting

Shell script to automatically download files

I am new to shell scripting and need to write a program to copy files that are posted as links on a specific url. I want all the links copied with the same file name and the one posted on the webpage containing the url onto a specific directory. That is the first part. The second part of the script... (2 Replies)
Discussion started by: libertyforall
2 Replies

7. Shell Programming and Scripting

Wget, download file from site's folder.

Ok, this is quite weird. wget -r mysite.com/mylink/ should get all the files recursively from the 'mylink' folder. The problem is that wget saves an index.html file! When I open this index.html with my browser I realize that it shows all the files in the current folder (plus an option to move... (3 Replies)
Discussion started by: hakermania
3 Replies

8. Shell Programming and Scripting

Shell Script for Upload/download files using cURL

hi please help me out here, i want to use curl command in shell script to test web pages, what i have is an opening page, when i click on a button on opening page, the next page comes up and then i have to upload a file n then click another button to submit and then comes the output page,... (2 Replies)
Discussion started by: Olivia
2 Replies

9. Shell Programming and Scripting

lftp script to connect to external sftp site and download to internal ftp and then send email

Hi there, I'm new to shell scripting and need some help if possible? I need to create a shell script (.sh) to run as a cron job on an ubuntu linux server to connect to an external sftp sites directory using credentials (which I have) and then download to our internal ftp server and then copy... (3 Replies)
Discussion started by: ghath
3 Replies

10. Shell Programming and Scripting

Download a site

A buddy of mine was telling me last night that you can write a bash script that will download an entire site in gedit?? Is this true??? I think I am going to fall in love with bash :D Any good tutorials?? (15 Replies)
Discussion started by: graphicsman
15 Replies
ED2K(1) 							  aMule utilities							   ED2K(1)

NAME
ed2k - aMule eD2k link parser SYNOPSIS
ed2k [-c <path>] [-t <num>] [-e] [-l] <eD2k-link> ed2k [-h] ed2k [-v] DESCRIPTION
Sends the given <eD2k-link> to aMule, i.e. writes it to the file ~/.aMule/ED2KLinks, which will be checked by aMule every second for links. [ -c <path>, --config-dir=<path> ] Read config from <path> instead of home [ -t, --category=<num> ] Set category for passed eD2k links to <num> [ -e, --emulecollection ] Loads all link found in the emulecollection given as <ed2k-link> [ -l, --list ] Lists all link found in the emulecollection given as <ed2k-link> [ -h, --help ] Prints a short usage description. [ -v, --version ] Displays the current version number. [ eD2k-link ] Adds an eD2k-link to the core. The eD2k link to be added can be: o a file link (ed2k://|file|...), it will be added to the download queue; o a server link (ed2k://|server|...), it will be added to the server list; o a serverlist link, in which case all servers in the list will be added to the server list; o a magnet link; o an emulecollection file. The order in which you give the parameters is important. You can give more than one link, and every link can have it's own params. For example ed2k <link1> -t2 <link2> will download <link1> in standard category and <link2> in category 2. FILES
~/.aMule/ED2KLinks REPORTING BUGS
Please report bugs either on our forum (http://forum.amule.org/), or in our bugtracker (http://bugs.amule.org/). Please do not report bugs in e-mail, neither to our mailing list nor directly to any team member. COPYRIGHT
aMule and all of its related utilities are distributed under the GNU General Public License. SEE ALSO
amule(1) AUTHOR
This manpage was written by Vollstrecker <amule@vollstreckernet.de> aMule eD2k link parser v1.5.1 November 2011 ED2K(1)
All times are GMT -4. The time now is 07:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy