Sponsored Content
Full Discussion: Download a site
Top Forums Shell Programming and Scripting Download a site Post 302721747 by graphicsman on Thursday 25th of October 2012 05:57:12 PM
Old 10-25-2012
ok ive tested each one on the site but how can I use this and make each one go into a folder or assign a name because they all go numerical by index.html??

Code:
Download Multiple Files / URLs Using Wget -i

First, store all the download files or URLs in a text file as:

$ cat > download-file-list.txt
URL1
URL2
URL3
URL4
Next, give the download-file-list.txt as argument to wget using -i option as shown below.

$ wget -i download-file-list.txt

I know there is a way to do it but for the life of me cannot remember how. Also how would I make it into a script to just call script and it would add them to an X folder? If you can explain it to me I would appreciate it.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

script for download files from ftp site

I'm new to scripting. I'm trying to write a script to download files from ftp site, the following is the script and the message i get after running the script. no files were downloaded :( Thanks advance! script: #!/usr/bin/ksh DAY=`date --date="-1 days" +%y%m%d` ftp -v -n "ftp.address" <<... (5 Replies)
Discussion started by: tiff-matt
5 Replies

2. Linux

shell script to download files from a site?

Hey everyone, my wife has purchased a bundle package of a bunch of images from a site, and now has to download each one of them manually. There are about 500 downloads, and it's quite a hassle to browse to each page and download them all individually. I would like to write a shell script to... (2 Replies)
Discussion started by: paqman
2 Replies

3. IP Networking

port access to site to site VPN

Setup a site to site VPN between two cisco routers. One of the site locations is unable to access ports such as https://example.com:9001 How do I let them go into port 9001? They can ssh, ftp, telnet and everything else. Is this a VPN issue or ACL access issue? I put permit ip host... (0 Replies)
Discussion started by: photon
0 Replies

4. Shell Programming and Scripting

Reconnect and download data from online site

Hi, I have to connect to online site and download some data from the online site but sometimes the site which i have to connect will be busy or it will not be working in such a case i have to connect atleast 3 times and try to download the data. I tried like this: use CGI::Carp... (2 Replies)
Discussion started by: vanitham
2 Replies

5. UNIX for Dummies Questions & Answers

Download image every 24hrs + put it in php site

Hi everyone, I am all new to this and I might not know what I am asking about :) I have access to a cPanel with cron tab scheduler. I know nothing re scripting creating php etc.. I own a ftp mainly for own email address, some storage, and a forum (ready-made one :P ) I have some knowledge... (5 Replies)
Discussion started by: zYx
5 Replies

6. Shell Programming and Scripting

Wget, download file from site's folder.

Ok, this is quite weird. wget -r mysite.com/mylink/ should get all the files recursively from the 'mylink' folder. The problem is that wget saves an index.html file! When I open this index.html with my browser I realize that it shows all the files in the current folder (plus an option to move... (3 Replies)
Discussion started by: hakermania
3 Replies

7. Shell Programming and Scripting

lftp script to connect to external sftp site and download to internal ftp and then send email

Hi there, I'm new to shell scripting and need some help if possible? I need to create a shell script (.sh) to run as a cron job on an ubuntu linux server to connect to an external sftp sites directory using credentials (which I have) and then download to our internal ftp server and then copy... (3 Replies)
Discussion started by: ghath
3 Replies

8. IP Networking

How to establish site to site vpn - Linux machine and cisco asa?

Hi, I am trying to establish vpn between my linux server and cisco asa at client side. I installed openswan on my cent os. Linux Server eth0 - 182.2.29.10 Gateway - 182.2.29.1 eth1 - 192.9.200.75 I have simple IPtables Like WAN="eth0" LAN="eth1" (0 Replies)
Discussion started by: ashokvpp
0 Replies

9. IP Networking

Does cisco 1921 router support site to site VPNs using IPSec?

Q: "Does Cisco 1921 router support,, act as an endpoint for, site to site VPNs using IPSec? If so, how many? " A: If you get the Cisco 1921/k9 with the security services bundle then it will have built in security features. Cisco, typically includes IP Sec tunnels I believe as part of that... (0 Replies)
Discussion started by: Ayaerlee
0 Replies

10. IP Networking

IPSec Openswan Site to Site VPN - Big Pain

Hi @all, I try to connect 2 LANs with IPSec/Openswan LAN 1: 192.168.0.0/24 LAN 2: 192.168.1.0/24 This is my Config: conn HomeVPN # # Left security gateway, subnet behind it, nexthop toward right. left=192.168.1.29 ... (1 Reply)
Discussion started by: bahnhasser83
1 Replies
SYSPROFILE(8)						      System Manager's Manual						     SYSPROFILE(8)

NAME
sysprofile - modular centralized shell configuration DESCRIPTION
sysprofile is a generic approach to configure shell settings in a modular and centralized way mostly aimed at avoiding work for lazy sysad- mins. It has only been tested to work with the bash shell. It basically consists of the small /etc/sysprofile shell script which invokes other small shell scripts having a .bash suffix which are contained in the /etc/sysprofile.d/ directory. The system administrator can drop in any script he wants without any naming convention other than that the scripts need to have a .bash suffix to enable automagic sourcing by /etc/sysprofile. This mechanism is set up by inserting a small shell routine into /etc/profile for login shells and optionally into /etc/bashrc and/or /etc/bash.bashrc for non-login shells from where the actual /etc/sysprofile script is invoked: if [ -f /etc/sysprofile ]; then . /etc/sysprofile fi For using "sysprofile" under X11, one can source it in a similar way from /etc/X11/Xsession or your X display manager's Xsession file to provide the same shell environment as under the console in X11. See the example files in /usr/share/doc/sysprofile/ for illustration. For usage of terminal emulators with a non-login bash shell under X11, take care to enable sysprofile via /etc/bash.bashrc. If not set this way, your terminal emulators won't come up with the environment defined by the scripts in /etc/sysprofile.d/. Users not wanting /etc/sysprofile to be sourced for their environment can easily disable it's automatic mechanism. It can be disabled by simply creating an empty file called $HOME/.nosysprofile in the user's home directory using e.g. the touch(1) command. Any single configuration file in /etc/sysprofile.d/ can be overridden by any user by creating a private $HOME/.sysprofile.d/ directory which may contain a user's own version of any configuration file to be sourced instead of the system default. It's names have just to match exactly the system's default /etc/sysprofile.d/ configuration files. Empty versions of these files contained in the $HOME/.syspro- file.d/ directory automatically disable sourcing of the system wide version. Naturally, users can add and include their own private script inventions to be automagically executed by /etc/sysprofile at login time. OPTIONS
There are no options other than those dictated by shell conventions. Anything is defined within the configuration scripts themselves. SEE ALSO
The README files and configuration examples contained in /etc/sysprofile.d/ and the manual pages bash(1), xdm(1x), xdm.options(5), and wdm(1x). Recommended further reading is everything related with shell programming. If you need a similar mechanism for executing code at logout time check out the related package syslogout(8) which is a very close compan- ion to sysprofile. BUGS
sysprofile in its current form is mainly restricted to bash(1) syntax. In fact it is actually a rather embarrassing quick and dirty hack than anything else - but it works. It serves the practical need to enable a centralized bash configuration until something better becomes available. Your constructive criticism in making this into something better" is very welcome. Before i forget to mention it: we take patches... ;-) AUTHOR
sysprofile was developed by Paul Seelig <pseelig@debian.org> specifically for the Debian GNU/Linux system. Feel free to port it to and use it anywhere else under the conditions of either the GNU public license or the BSD license or both. Better yet, please help to make it into something more worthwhile than it currently is. SYSPROFILE(8)
All times are GMT -4. The time now is 10:20 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy