09-11-2013
Quote:
Originally Posted by
ajayram_arya
there are only 80 files on that link which i am downloading . Some files come early and some late . So till i get all the files i wanted to loop in till I get all the files.
Why not use wget's retry options?
-t 20 to try 20 times before giving up.
Otherwise you're stuck with trying to parse an HTML file and figure out which things wget missed.
10 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
I need to download some files from a remote server using ftp. I have ftp'd into the site. I then do an mget * to retrieve all of the data files. Everything seems to proceed normally and I am given feedback that the files were downloaded. Now if I go into the DOS Shell or Windows explorer, it list... (5 Replies)
Discussion started by: ralphisnow
5 Replies
2. UNIX for Dummies Questions & Answers
Guys - I am new to Unix scripting and am in need for a script that does the following. I have bits and pieces created and tested but i am just having a little difficult time getting it all together.
- Loop through till it finds a string in a specific file.
Any help is greatly appreciated.
... (1 Reply)
Discussion started by: mrehman
1 Replies
3. Shell Programming and Scripting
Dear All
I have a file like this
112534554
446538656
444695656
225696966
226569744
228787874
113536566
443533535
222564552
115464656
225445345
225533234
I want to cut the file into different parts where the first two columns are '11' . The first two columns will be either... (3 Replies)
Discussion started by: anoopvraj
3 Replies
4. Shell Programming and Scripting
I am trying to download a page and retrieve only wav and mp3 files via wget.
the website is:
Alarm Sounds | Free Sound Effects | Alarm Sound Clips | Sound Bites
my command is :
wget -rl 2 -e robots=off -A wav,mp3 http://soundbible.com/tags-alarm.html
When not using the -A wav,mp3... (2 Replies)
Discussion started by: Narnie
2 Replies
5. Shell Programming and Scripting
Hi everybody, I would greatly appreciate some expertise in this matter. I am trying find an efficient way to batch download files from a website and rename each file with the url it originated from (from the CLI). (ie. Instead of xyz.zip, the output file would be http://www.abc.com/xyz.zip) A... (10 Replies)
Discussion started by: o0110o
10 Replies
6. Web Development
Hello,
I have setup Cherokee web server and php 5.2 in Opensolaris zone. Problem is that all .php files are downloaded from web server and not served when I use IP address instead of DNS name in web brovser.
Example: test.mydomain.com <-- php works
192.168.0.10/index.php <--... (3 Replies)
Discussion started by: kreno
3 Replies
7. Shell Programming and Scripting
hello.
How can I detect within script, that the downloaded file had not a correct size.
linux:~ # wget --limit-rate=20k --ignore-length -O /Software_Downloaded/MULTIMEDIA_ADDON/skype-4.1.0.20-suse.i586.rpm ... (6 Replies)
Discussion started by: jcdole
6 Replies
8. Shell Programming and Scripting
I need help to put a script where it runs the svn command grep'ing for the ticket# in the comments to see if the ticket was used in the latest commit.
so on command line:
./test.sh ticket-1 ticket-2 ticket-3
It should be able to check if ticket-1 is used first and if not then check if... (2 Replies)
Discussion started by: iaav
2 Replies
9. Shell Programming and Scripting
I have the following code:
# Get the line of stations_info.txt starting with "${xstation1} " and copy it to file temp.txt
grep "^${xstation1} " stations_info.txt > temp.txt
# Get lat and long of station
nl=0
... (2 Replies)
Discussion started by: claire.a
2 Replies
10. Shell Programming and Scripting
Hello,
I have a server that I have to ftp files off and they all start SGRD and are followed by 6 numbers.
SGRD000001
SGRD000002
SGRD000003
The script I have will run every 10 mins to pick up files as new ones will be coming in all the time and what I want to do is delete the files I have... (7 Replies)
Discussion started by: sph90457
7 Replies
LEARN ABOUT DEBIAN
hxincl
HXINCL(1) HTML-XML-utils HXINCL(1)
NAME
hxincl - expand included HTML or XML files
SYNOPSIS
hxincl [ -x ] [ -f ] [ -s name=subst ] [ -s name=subst ]... [ -b base ] [ file-or-URL ]
DESCRIPTION
The hxincl command copies an HTML or XML file to standard output, looking for comments with a certain structure. Such a comment is
replaced by the file whose name is given as the attribute of the directive. For example:
...<!-- include "foo.html" -->...
will be replaced by the content of the file foo.html. It is important to note that you must quote filenames if they contain white space.
The comment is replaced by
<!-- begin-include "foo.html" -->
before the included text and
<!-- end-include "foo.html" -->
after it. These comments make it possible to run hxincl on the resulting file again to update the inclusions.
Single quotes are allowed instead of double quotes. And if the file name contains no spaces, the quotes may also be omitted.
OPTIONS
The following options are supported:
-x Use XML conventions: empty elements are written with a slash at the end: <IMG />.
-b base Sets the base URL for resolving relative URLs. By default the file given as argument is the base URL.
-f Removes the comments after including the files. This means hxincl connot be run on the resulting file later to update the inclu-
sions. (Mnemonic: final or frozen.)
-s name=substitution
Include a different file than the one mentioned in the directive. If the comment is
<!-- include "name" -->
the file substitution is included instead. The option -s may occur multiple times.
OPERANDS
The following operand is supported:
file-or-URL
The name of an HTML or XML file or the URL of one. If absent, standard input is read instead.
EXIT STATUS
The following exit values are returned:
0 Successful completion.
> 0 An error occurred in the parsing of one of the HTML or XML files.
ENVIRONMENT
To use a proxy to retrieve remote files, set the environment variables http_proxy or ftp_proxy. E.g., http_proxy="http://localhost:8080/"
BUGS
Assumes UTF-8 as input. Doesn't expand character entities. Instead pipe the input through hxunent(1) and asc2xml(1) to convert it to UTF-8.
Remote files (specified with a URL) are currently only supported for HTTP. Password-protected files or files that depend on HTTP "cookies"
are not handled. (You can use tools such as curl(1) or wget(1) to retrieve such files.)
SEE ALSO
asc2xml(1), hxnormalize(1), hxnum(1), hxprune(1), hxtoc(1), hxunent(1), xml2asc(1), UTF-8 (RFC 2279)
6.x 10 Jul 2011 HXINCL(1)