Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Random web page download wget script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Random web page download wget script
# 8  
Old 04-20-2013
Works great!! I amended it to include my very simple additions.

Code:
#!/bin/bash
url_list=( http://www.bbc.co.uk http://www.cnn.com http://www.msn.com )

#Number of users to mimic simultaneously
users=20

function one_user () {
  local user=$1
  while [ 1 -eq 1 ]; do
    local wait=`expr $RANDOM % 120 + 30`
    local n=`expr $RANDOM % 3`
    local url=${url_list[$n]}
    time=`date +"%T"`
    date=`date +"%m-%d-%y"`
    wget=`wget -E -H -T 30 -k -K -p --delete-after --no-cache -e robots=off $url 2>&1 | grep Downloaded | awk -F " " '{print $6}'`
    echo $date,$time,client$user,$url,$wget
    # echo user = $user wait = $wait url = $url
    sleep $wait
  done
  }

for (( user = 1; user <= $users; user++ )); do
  one_user $user &
done

Sample output.

Code:
04-21-13,00:34:01,client1,http://www.msn.com,1.8s
04-21-13,00:34:01,client14,http://www.bbc.co.uk,3.6s
04-21-13,00:34:34,client2,http://www.msn.com,1.5s
04-21-13,00:34:39,client19,http://www.msn.com,1.7s
04-21-13,00:34:34,client12,http://www.bbc.co.uk,3.6s
04-21-13,00:34:34,client4,http://www.cnn.com,4.9s
04-21-13,00:34:40,client20,http://www.bbc.co.uk,3.4s
04-21-13,00:34:49,client11,http://www.msn.com,1.9s
04-21-13,00:34:58,client14,http://www.bbc.co.uk,0.9s
04-21-13,00:34:50,client8,http://www.bbc.co.uk,3.8s
04-21-13,00:34:58,client5,http://www.bbc.co.uk,3.6s
04-21-13,00:35:19,client12,http://www.msn.com,1.4s
04-21-13,00:35:25,client10,http://www.msn.com,1.5s
04-21-13,00:35:20,client13,http://www.bbc.co.uk,3.3s
04-21-13,00:35:29,client3,http://www.bbc.co.uk,3.1s
04-21-13,00:35:35,client8,http://www.bbc.co.uk,3.1s
04-21-13,00:35:46,client9,http://www.msn.com,1.4s
04-21-13,00:35:55,client17,http://www.msn.com,2.1s
04-21-13,00:35:58,client7,http://www.msn.com,1.4s
04-21-13,00:35:50,client18,http://www.cnn.com,4.4s

Much appreciated hanson44!!
# 9  
Old 04-20-2013
That's really a great testing system you have. It seems like it will be very interesting to observe how it behaves, as it mimics those downloading users.

Back to the script, the only (perhaps nit-picky) thing I would suggest would be to put the wget arguments into a separate variable to improve readability:
Code:
args="-E -H -T 30 -k -K -p --delete-after --no-cache -e robots=off"
wget=`wget $args $url 2>&1 | grep Downloaded | awk -F " " '{print $6}'`

# 10  
Old 04-20-2013
Thanks Hanson, much of it was your work.

I'm using it to differentiate between two mobile systems although it could be used for stress testing.

Another useful addition would be to include some file downloads, say 10MB. However, I just tested this and the grep in the script wouldn't work if i included the file URLs. In the example below, i would need the info "5.7s". Any idea?

Code:
[root@scripts]# wget http://download.thinkbroadband.com/5MB.zip
--2013-04-21 01:32:47--  http://download.thinkbroadband.com/5MB.zip
Resolving download.thinkbroadband.com... 80.249.99.148, 2a02:68:1:7::1
Connecting to download.thinkbroadband.com|80.249.99.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5242880 (5.0M) [application/zip]
Saving to: `5MB.zip'

100%[==============================================================================================================================>] 5,242,880   1.34M/s   in 5.7s

2013-04-21 01:32:54 (900 KB/s) - `5MB.zip' saved [5242880/5242880]

---------- Post updated at 08:01 PM ---------- Previous update was at 07:38 PM ----------

Ah, figured it out. I used the last occurrence of "in" as the target as thats common across wget web page and file downloads.

Code:
#!/bin/bash
url_list=( http://www.bbc.co.uk http://www.cnn.com http://www.msn.com )

#Number of users to mimic simultaneously
users=20

#wget arguments
args="-E -H -T 30 -k -K -p --delete-after --no-cache -e robots=off"

function one_user () {
  local user=$1
  while [ 1 -eq 1 ]; do
    local wait=`expr $RANDOM % 120 + 30`
    local n=`expr $RANDOM % 3`
    local url=${url_list[$n]}
    time=`date +"%T"`
    date=`date +"%m-%d-%y"`
    wget=`wget $args $url 2>&1 | awk '/in/{a=$0}END{print a}' | awk -F "in" '{print$2}'`
    echo $date,$time,client$user,$url,$wget
    # echo user = $user wait = $wait url = $url
    sleep $wait
  done
  }

for (( user = 1; user <= $users; user++ )); do
  one_user $user &
done

Last edited by shadyuk; 04-20-2013 at 09:45 PM..
# 11  
Old 04-21-2013
As Scott write, why do you still use back tics `` and expr
You code looks better without.

Code:
#!/bin/bash
url_list=( http://www.bbc.co.uk http://www.cnn.com http://www.msn.com )

#Number of users to mimic simultaneously
users=20

#wget arguments
args="-E -H -T 30 -k -K -p --delete-after --no-cache -e robots=off"

function one_user () {
  local user=$1
  while [ 1 -eq 1 ]; do
    local wait=$((RANDOM % 120 + 30))
    local n=$((RANDOM % 3))
    local url=${url_list[$n]}
    time=$(date +"%T")
    date=$(date +"%m-%d-%y")
    wget=$(wget $args $url 2>&1 | awk '/in/{a=$0}END{print a}' | awk -F "in" '{print$2}')
    echo $date,$time,client$user,$url,$wget
    # echo user = $user wait = $wait url = $url
    sleep $wait
  done
  }

for (( user = 1; user <= $users; user++ )); do
  one_user $user &
done

# 12  
Old 04-21-2013
Code:
why do you still use back tics  ``  and  expr 
You code looks better without.

I would suggest that either way is fine.
# 13  
Old 04-21-2013
Either way do work but backticks are easy to miss since they are so small ` and nesting is not easy with them.
BashFAQ/082 - Greg's Wiki
# 14  
Old 04-21-2013
Thanks for sending the reference. We're in basic agreement.

I agree $( ) is better for nested commands. But I think it's bad practice to nest commands. Seems much clearer to use an intermediate variable.

I perhaps agree with the example about backslashes. But this seems pretty rare. And neither format seems "obvious". Multiple backslashes are confusing to most people, including me.

For "nested quoting", I didn't realize there was a difference between the shells. I use bash, so would not matter. But I agree this case is clearer with the $( ) syntax for compatibility with sh and ksh.

You're right the backticks are tiny. But for some reason, they are easy to see for me and still leave the code clean. The article says "easily confused with a single quote". Sounds reasonable. But that has NEVER happened to me. Maybe I'm just used to those backticks, from repeated usage...

If the author of the article were really objective, they would say backticks take one less space. Yes, it's nit-picky. But so are the other points.

BTW, I have nothing against the $( ) syntax. It's perfectly fine. I just don't think it matters in the great majority of cases. I see these "crusades" (I am NOT referring to any posts here) against certain practices, and think they are a tempest in a teapot, because the practices seem useful and easy to understand to me. Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to use cURL to download web page with authentification (form)?

Hello, I'm new in the forum and really beginer, and also sorry form my bad english. I use linux and want to create little program to download automaticaly some pdf (invoices) and put in a folder of my computer. I learn how to do and programme with ubuntu but the program will be implemented... (1 Reply)
Discussion started by: MarcelOrMittal
1 Replies

2. Shell Programming and Scripting

Refresh web page in bash script

hello, I am trying to refresh my web page which is created in bash script. I have a HTML page which when press a button calls a bash script. this bash script created the same page with dynamic data. When pressing the button I am calling to a function that set time out of 7 seconds and and after... (1 Reply)
Discussion started by: SH78
1 Replies

3. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403... (2 Replies)
Discussion started by: Prasannag87
2 Replies

4. UNIX for Dummies Questions & Answers

List and download web page files

Hello, Does anyone know of a way to list all files related to a single web page and then to download say 4 files at a time simultaneously until the entire web page has been downloaded successfully? I'm essentially trying to mimic a browser. Thanks. (2 Replies)
Discussion started by: shadyuk
2 Replies

5. HP-UX

Help running a unix script from a web page

First, let me state that I am completely out of my realm with this. I have a server running HPUX. I'm not even sure if this can be considered a UNIX question and for that let me apologize in advance. I need to create a web page where a client can input 2 variables (i.e. date and phone number).... (0 Replies)
Discussion started by: grinds
0 Replies

6. UNIX for Dummies Questions & Answers

Possible to download web page's text to a file?

Hi, Say there is a web page that contains just text only - that is, even the source code is just the text itself, nothing more. An example would be "http://mynasadata.larc.nasa.gov/docs/ocean_percent.txt" Is there a UNIX command that would allow me to download this text and store it in a... (1 Reply)
Discussion started by: Breanne
1 Replies

7. Shell Programming and Scripting

Perl script to copy contents of a web page

Hi All, Sorry to ask this question and i am not sure whether it is possible. please reply to my question. Thanks in advance. I need a perl script ( or any linux compatible scripts ) to copy the graphical contents of the webpage to a word pad. Say for example, i have a documentation site... (10 Replies)
Discussion started by: anand.linux1984
10 Replies

8. Shell Programming and Scripting

how to redirect to a web-page by shell script

Dear all, I am calling a korn shell script(CGI script) by a web-page. This shell script do some checking in a unix file and return true or false. Now within the same script, If it returns true then I want to redirect to another web-page stored in htdocs directory. Example: Login page sends a... (3 Replies)
Discussion started by: ravi18s
3 Replies

9. Shell Programming and Scripting

Script to download file using wget

Hi I need a Shell script that will download a text file every second from a http server using wget. Can anyone provide me any pointers or sample scripts that will help me go about this task ??? regards techie (1 Reply)
Discussion started by: techie82
1 Replies

10. Shell Programming and Scripting

running shell script thru WEB page ....

....passing variable via list... here 's the HTML code extract : **************** <form method=post action=http://servername/cgi-bin/cgi-comptage_diff.ksh> <table border...........> .............. </table> <table bgcolor=#FFFFFF width="980"> ... (6 Replies)
Discussion started by: Nicol
6 Replies
Login or Register to Ask a Question