06-16-2014
If it's saving index.html, you forgot the --spider.
You can feed wget a list of URL's with awk '{...}' | wget -I - ...
This User Gave Thanks to Corona688 For This Post:
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I'm running Fedora Core 6 as an FTP server on a powerMac G4...
I'm trying to create a script to remove files older than 3 days...
I'm able to find all data older than 3 days but it finds hidden files such as
/home/ftp/goossens/.canna
/home/ftp/goossens/.kde... (4 Replies)
Discussion started by: James_UK
4 Replies
2. Solaris
On Solaris, suppose there is a directory 'dir'.
Log files of size approx 1MB are continuously being
deposited here by scp command. I have a script that scans
this dir every 5 mins and moves away the log files that
have been deposited so far.
How do I design my script so that I pick up *only*... (6 Replies)
Discussion started by: sentak
6 Replies
3. UNIX for Advanced & Expert Users
I'm using wget 1.11.4 on Cygwin 1.5.25.
I'm trying to recursively download a directory tree, which is the root of a javadoc tree.
This is approximately the command line I tried:
wget -x -p -r http://<host>/.../apidoc
When it finished, it seemed like it downloaded... (0 Replies)
Discussion started by: dkarr
0 Replies
4. Shell Programming and Scripting
Can you tell me how to download the directory tree just starting from "project1/" in this URL?
"https://somesite.com/projects/t/project1/"
This command does not seem to do what I want as it downloads also files from the upper hierarchy:
wget --no-check-certificate --http-user=user... (4 Replies)
Discussion started by: majormark
4 Replies
5. Shell Programming and Scripting
Is there a way to customize ls to ignore files ending with ~ and #? (those are Emacs backup and auto-save files). I found -B option, which only ignores ~ files (2 Replies)
Discussion started by: yaroslavvb
2 Replies
6. Shell Programming and Scripting
Hello,
I know find can be prevented from recursing into directories with something like the following...
find . -name .svn -prune -a type d
But how can I completely prevent directories of a certain name (.svn) from being displayed at all, the top level and the children?
I really... (2 Replies)
Discussion started by: nwb123
2 Replies
7. Shell Programming and Scripting
Hello Unix Geeks,
I am in a situation to use wget for crawling a site where the site contains 5 IP addresses. Out of 5, 4 are accessible and 1 is having a problem due to firewall problems.
In this case, my wget is getting stuck with that X.X.X.X and giving up. How can I ignore this IP and... (4 Replies)
Discussion started by: sathyaonnuix
4 Replies
8. Shell Programming and Scripting
Dear All,
I am using find command
find /my_rep/*/RKYPROOF/*/*/WDM/HOME_INT/PWD_DATA -name rk*myguidelines*.pdf -print
The problem i am facing here is find /my_rep/*/
the directory after my_rep could be mice001, mice002 and mice001_PO, mice002_PO
i want to ignore mice***_PO directory... (3 Replies)
Discussion started by: yadavricky
3 Replies
9. Shell Programming and Scripting
i have a cron that mirrors a site periodically
wget -r -nc --passive-ftp ftp://user:pass@123.456.789.0
i want to download this into a directory called /files
but when I do this, it always create a new directory called "123.456.789.0" (the hostname)
it puts it into /files/123.456.789.0
but... (3 Replies)
Discussion started by: vanessafan99
3 Replies
10. UNIX for Advanced & Expert Users
I am using aix. I would like to ignore the /u directory. I tried this but it is not working.
find / -type f -type d \( -path /u \) -prune -o -name '*rpm*' 2>/dev/null
/u/appx/ls.rpm
/u/arch/vim.rpm (4 Replies)
Discussion started by: cokedude
4 Replies
LEARN ABOUT DEBIAN
jigdo-lite
JIGDO-LITE(1) JIGDO-LITE(1)
NAME
jigdo-lite - Download jigdo files using wget
SYNOPSIS
jigdo-lite [ URL ]
DESCRIPTION
See jigdo-file(1) for an introduction to Jigsaw Download.
Given the URL of a `.jigdo' file, jigdo-lite downloads the large file (e.g. a CD image) that has been made available through that URL.
wget(1) is used to download the necessary pieces of administrative data (contained in the `.jigdo' file and a corresponding `.template'
file) as well as the many pieces that the large file is made from. The jigdo-file(1) utility is used to reconstruct the large file from the
pieces.
`.jigdo' files that contain references to Debian mirrors are treated specially: When such a file is recognized, you are asked to select one
mirror out of a list of all Debian mirrors.
If URL is not given on the command line, the script prompts for a location to download the `.jigdo' file from. The following command line
options are recognized:
-h --help
Output short summary of command syntax.
-v --version
Output version number.
--scan FILES
Do not ask for "Files to scan", use this path.
--noask
Do not ask any questions, instead behave as if the user had pressed Return at all prompts. This can be useful when running jigdo-
lite from cron jobs or in other non-interactive environments.
SEE ALSO
jigdo-file(1), jigdo-mirror(1), wget(1) (or `info wget')
CD images for Debian Linux can be downloaded with jigdo <URL:http://www.debian.org/CD/jigdo-cd/>.
AUTHOR
Jigsaw Download <URL:http://atterer.net/jigdo/> was written by Richard Atterer <jigdo atterer.net>, to make downloading of CD ROM images
for the Debian Linux distribution more convenient.
19 May 2006 JIGDO-LITE(1)