Download a site


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Download a site
# 1  
Old 10-12-2012
Download a site

A buddy of mine was telling me last night that you can write a bash script that will download an entire site in gedit?? Is this true??? I think I am going to fall in love with bash Smilie Any good tutorials??
# 2  
Old 10-12-2012
Quote:
Originally Posted by graphicsman
A buddy of mine was telling me last night that you can write a bash script that will download an entire site in gedit?? Is this true??? I think I am going to fall in love with bash Smilie Any good tutorials??
Try to read about wget...Smilie
This User Gave Thanks to pamu For This Post:
# 3  
Old 10-12-2012
Quote:
Originally Posted by pamu
Try to read about wget...Smilie
my god where has this been all my life... Smilie

I think I shall do some damage, lol.


EDIT:
so basically the many times ive spent downloading things one by one from sites I could have done this in 5 mins and been done?? I would assume I can download certain elements of a full site then.

Last edited by graphicsman; 10-12-2012 at 12:09 PM..
# 4  
Old 10-12-2012
I'm not sure what you mean.
# 5  
Old 10-12-2012
well an example. I like going to font sites and downloading useful fonts but if I learn how to write a wget I could download everything from a site and use my font manager on my rip station to use the fonts. So i could then change an hour downloading fonts to use I can use wget and download a whole site's fonts in a couple of seconds..
# 6  
Old 10-12-2012
What I sometimes do is extract all links from a page by
1) wget -k a particular page
2) egrep -o "http://[^"']+" resulting-filename.html | grep "stuff-I-want" > newlist
3) wget -I newlist

It won't work for annoying javascript or form-based downloads.
# 7  
Old 10-12-2012
Quote:
Originally Posted by graphicsman
well an example. I like going to font sites and downloading useful fonts but if I learn how to write a wget I could download everything from a site and use my font manager on my rip station to use the fonts. So i could then change an hour downloading fonts to use I can use wget and download a whole site's fonts in a couple of seconds..
I don't think what you are trying to achieve is that much easy to do with wget.

You should take look at the Site structure. How the site is designed. Normally all the download items, files are saved in different location and site is just using the link to show them and can be downloaded as per user's request from particular form.(I am talking about java get methods).
And site is normally hosted on different server.
So what you are trying to achieve is I think not possible.
You can't download all the font's like this. If you know the link of of all the files then it can be done using wget ..Smilie


Please see this (link removed)
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. IP Networking

IPSec Openswan Site to Site VPN - Big Pain

Hi @all, I try to connect 2 LANs with IPSec/Openswan LAN 1: 192.168.0.0/24 LAN 2: 192.168.1.0/24 This is my Config: conn HomeVPN # # Left security gateway, subnet behind it, nexthop toward right. left=192.168.1.29 ... (1 Reply)
Discussion started by: bahnhasser83
1 Replies

2. IP Networking

Does cisco 1921 router support site to site VPNs using IPSec?

Q: "Does Cisco 1921 router support,, act as an endpoint for, site to site VPNs using IPSec? If so, how many? " A: If you get the Cisco 1921/k9 with the security services bundle then it will have built in security features. Cisco, typically includes IP Sec tunnels I believe as part of that... (0 Replies)
Discussion started by: Ayaerlee
0 Replies

3. IP Networking

How to establish site to site vpn - Linux machine and cisco asa?

Hi, I am trying to establish vpn between my linux server and cisco asa at client side. I installed openswan on my cent os. Linux Server eth0 - 182.2.29.10 Gateway - 182.2.29.1 eth1 - 192.9.200.75 I have simple IPtables Like WAN="eth0" LAN="eth1" (0 Replies)
Discussion started by: ashokvpp
0 Replies

4. Shell Programming and Scripting

lftp script to connect to external sftp site and download to internal ftp and then send email

Hi there, I'm new to shell scripting and need some help if possible? I need to create a shell script (.sh) to run as a cron job on an ubuntu linux server to connect to an external sftp sites directory using credentials (which I have) and then download to our internal ftp server and then copy... (3 Replies)
Discussion started by: ghath
3 Replies

5. Shell Programming and Scripting

Wget, download file from site's folder.

Ok, this is quite weird. wget -r mysite.com/mylink/ should get all the files recursively from the 'mylink' folder. The problem is that wget saves an index.html file! When I open this index.html with my browser I realize that it shows all the files in the current folder (plus an option to move... (3 Replies)
Discussion started by: hakermania
3 Replies

6. UNIX for Dummies Questions & Answers

Download image every 24hrs + put it in php site

Hi everyone, I am all new to this and I might not know what I am asking about :) I have access to a cPanel with cron tab scheduler. I know nothing re scripting creating php etc.. I own a ftp mainly for own email address, some storage, and a forum (ready-made one :P ) I have some knowledge... (5 Replies)
Discussion started by: zYx
5 Replies

7. Shell Programming and Scripting

Reconnect and download data from online site

Hi, I have to connect to online site and download some data from the online site but sometimes the site which i have to connect will be busy or it will not be working in such a case i have to connect atleast 3 times and try to download the data. I tried like this: use CGI::Carp... (2 Replies)
Discussion started by: vanitham
2 Replies

8. IP Networking

port access to site to site VPN

Setup a site to site VPN between two cisco routers. One of the site locations is unable to access ports such as https://example.com:9001 How do I let them go into port 9001? They can ssh, ftp, telnet and everything else. Is this a VPN issue or ACL access issue? I put permit ip host... (0 Replies)
Discussion started by: photon
0 Replies

9. Linux

shell script to download files from a site?

Hey everyone, my wife has purchased a bundle package of a bunch of images from a site, and now has to download each one of them manually. There are about 500 downloads, and it's quite a hassle to browse to each page and download them all individually. I would like to write a shell script to... (2 Replies)
Discussion started by: paqman
2 Replies

10. Shell Programming and Scripting

script for download files from ftp site

I'm new to scripting. I'm trying to write a script to download files from ftp site, the following is the script and the message i get after running the script. no files were downloaded :( Thanks advance! script: #!/usr/bin/ksh DAY=`date --date="-1 days" +%y%m%d` ftp -v -n "ftp.address" <<... (5 Replies)
Discussion started by: tiff-matt
5 Replies
Login or Register to Ask a Question