Good use of while read. You can redirect the entire loop instead of reopening download.txt 1000 times though:
wget also has some features which make a loop unnecessary though
wget is able to read a list of files with -i. The -nv option is also useful, to make it still print completed files without printing all the complicated junk wget usually does.
This should be much faster than calling wget 1000 times since it is able to re-use the same connection if it's connecting to the same site. Concurrency may not be necessary ( and may not be desirable in many cases -- how fast is your connection? ) but if it is, I'd split the list into parts and use wget -i on those parts.
Good day!
I am trying to learn how to use the "sed" editor, to perform multiple edits on multiple files in multiple directories.
I have one script that tries to call up each file and process it according to the edits listed in a second script. I am using a small input text to test these, at... (12 Replies)
Dear Experts
Why we always hear that unix operating system is Multi User and Multi task. What does these two means. I have looked at some books and documents but couldn't find aclear explenation. Can we say Windows operating system is also multi user and multi task??
Thanks for your help in... (6 Replies)
Hello,
I need "devices.common.IBM.ml 1.4.0.0 C F Multi Link Interface Runtime" to be installed on my machine.
I need it for two SAN cards to work correctly.
Where do I get it ?
thanks
Vilius (1 Reply)
Hi there..
I need a proper "mutt" command to send a mail with html body and html attachment at a time.
Also if possible let me know the other commands to do this task.
Please help me.. (2 Replies)
Hi,
I have a problem where I need to make this input:
nameRow1a,text1a,text2a,floatValue1a,FloatValue2a,...,floatValue140a
nameRow1b,text1b,text2b,floatValue1b,FloatValue2b,...,floatValue140b
look like this output:
nameRow1a,text1b,text2a,(floatValue1a - floatValue1b),(floatValue2a -... (4 Replies)
Hi all:
Been racking my brain on this for the last couple of days and what has been most frustrating is that this is the last piece I need to complete a project.
There are numerous posts discussing mutt in this forum and others but I have been unable to find similar issues.
Running with... (1 Reply)
I've an HTML page where the pie chart is generated with google java code with the required input values in UNIX.
The HMTL page is generated in UNIX and then when it loads in browser, the code is interpreted thought internet and the pie chart is generated. This is done by the java code in the... (4 Replies)
I am trying to write a large X app. I have successfully modified my xorg.conf to setup 4 monitors on an NVIDIA Quatro5200. I am trying to modify a simple hello world application to open a window on three of the four monitors. depending on the changes to loop the window creation section and event... (2 Replies)
httpindex(1) General Commands Manual httpindex(1)NAME
httpindex - HTTP front-end for SWISH++ indexer
SYNOPSIS
wget [ options ] URL... 2>&1 | httpindex [ options ]
DESCRIPTION
httpindex is a front-end for index++(1) to index files copied from remote servers using wget(1). The files (in a copy of the remote direc-
tory structure) can be kept, deleted, or replaced with their descriptions after indexing.
OPTIONS
wget Options
The wget(1) options that are required are: -A, -nv, -r, and -x; the ones that are highly recommended are: -l, -nh, -t, and -w. (See the
EXAMPLE.)
httpindex Options
httpindex accepts the same short options as index++(1) except for -H, -I, -l, -r, -S, and -V.
The following options are unique to httpindex:
-d Replace the text of local copies of retrieved files with their descriptions after they have been indexed. This is useful to display
file descriptions in search results without having to have complete copies of the remote files thus saving filesystem space. (See
the extract_description() function in WWW(3) for details about how descriptions are extracted.)
-D Delete the local copies of retrieved files after they have been indexed. This prevents your local filesystem from filling up with
copies of remote files.
EXAMPLE
To index all HTML and text files on a remote web server keeping descriptions locally:
wget -A html,txt -linf -t2 -rxnv -nh -w2 http://www.foo.com 2>&1 |
httpindex -d -e'html:*.html,text:*.txt'
Note that you need to redirect wget(1)'s output from standard error to standard output in order to pipe it to httpindex.
EXIT STATUS
Exits with a value of zero only if indexing completed sucessfully; non-zero otherwise.
CAVEATS
In addition to those for index++(1), httpindex does not correctly handle the use of multiple -e, -E, -m, or -M options (because the Perl
script uses the standard GetOpt::Std package for processing command-line options that doesn't). The last of any of those options ``wins.''
The work-around is to use multiple values for those options seperated by commas to a single one of those options. For example, if you want
to do:
httpindex -e'html:*.html' -e'text:*.txt'
do this instead:
httpindex -e'html:*.html,text:*.txt'
SEE ALSO
index++(1), wget(1), WWW(3)AUTHOR
Paul J. Lucas <pauljlucas@mac.com>
SWISH++ August 2, 2005 httpindex(1)