List and download web page files


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers List and download web page files
# 1  
Old 07-05-2013
List and download web page files

Hello,

Does anyone know of a way to list all files related to a single web page and then to download say 4 files at a time simultaneously until the entire web page has been downloaded successfully? I'm essentially trying to mimic a browser.

Thanks.
# 2  
Old 07-07-2013
Have you considered using wget or curl ?
# 3  
Old 07-18-2013
I've tried wget but this downloads one file at a time. I need a way of listing all the files for a web page and then scripting (in bash) a process to download say four of them simultaneously and to go through the list till page dl completion. I then want to return the amount of time it took to download all the files for that page.

Any guidance appreciated as the bit im struggling with is the listing of files.

Thanks
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to use cURL to download web page with authentification (form)?

Hello, I'm new in the forum and really beginer, and also sorry form my bad english. I use linux and want to create little program to download automaticaly some pdf (invoices) and put in a folder of my computer. I learn how to do and programme with ubuntu but the program will be implemented... (1 Reply)
Discussion started by: MarcelOrMittal
1 Replies

2. Web Development

List items not arranged as expected on web page

greetings, i have a list of items that are picks to be arranged within a row that is 22 in height. this "plugin" looks fine in the old installation but after installing a new version of the web app and dropping this "plugin" in place the layout isn't as i expect. they are arranged vertically... (1 Reply)
Discussion started by: crimso
1 Replies

3. Shell Programming and Scripting

Download dynamic generated image from HTML page

I've an HTML page where the pie chart is generated with google java code with the required input values in UNIX. The HMTL page is generated in UNIX and then when it loads in browser, the code is interpreted thought internet and the pie chart is generated. This is done by the java code in the... (4 Replies)
Discussion started by: Amutha
4 Replies

4. Shell Programming and Scripting

Random web page download wget script

Hi, I've been attempting to create a script that downloads web pages at random intervals to mimic typical user usage. However I'm struggling to link $url to the URL list and thus wget complains of a missing URL. Any ideas? Thanks #!/bin/sh #URL List url1="http://www.bbc.co.uk"... (14 Replies)
Discussion started by: shadyuk
14 Replies

5. UNIX for Dummies Questions & Answers

Possible to download web page's text to a file?

Hi, Say there is a web page that contains just text only - that is, even the source code is just the text itself, nothing more. An example would be "http://mynasadata.larc.nasa.gov/docs/ocean_percent.txt" Is there a UNIX command that would allow me to download this text and store it in a... (1 Reply)
Discussion started by: Breanne
1 Replies

6. AIX

nmon and web page !

nmon and web page ! Is there any way to let nmon be configured with external Web Page and updating the same web page to be graphic monitoring. Pls advice ... (1 Reply)
Discussion started by: Mr.AIX
1 Replies

7. UNIX for Dummies Questions & Answers

Looking for a web page that won't let me in

Hi, I have a project for school using wget and egrep to locate pattern locations on a web page. One of the things we have to do is handle an "access denied" exception. Here is the problem, I can not think of/find any web pages that give me an access denied error to play with, can anyone suggest... (1 Reply)
Discussion started by: njmiano
1 Replies

8. Shell Programming and Scripting

Printing to a web page

I have a shell script that runs periodic upgrades on machines. I want to print certain echo commands from the shell script onto a webpage. What command in shell should I use for doing this. (1 Reply)
Discussion started by: lassimanji
1 Replies

9. Shell Programming and Scripting

Web content download

There is a web API https://pi.ercot.com/contentproxy/publicList?folder_id=10001937 As soon as this is fired on the browser a page comes in which many files are listed for downloads I want to download all the files ....is it possible to write a shell script which can download the files (1 Reply)
Discussion started by: viv1
1 Replies

10. Shell Programming and Scripting

Want to show files on web page

hello Unix guru i want to show performance results on my internal website . We are manitaing the site through Wiki . I have to add new page in that . can someone help me to write shell script for that .. i want to display files datewise . my files names are starting with date . if... (3 Replies)
Discussion started by: deepa20
3 Replies
Login or Register to Ask a Question