Sponsored Content
Top Forums Shell Programming and Scripting Random web page download wget script Post 302796803 by shadyuk on Saturday 20th of April 2013 09:01:15 PM
Old 04-20-2013
Thanks Hanson, much of it was your work.

I'm using it to differentiate between two mobile systems although it could be used for stress testing.

Another useful addition would be to include some file downloads, say 10MB. However, I just tested this and the grep in the script wouldn't work if i included the file URLs. In the example below, i would need the info "5.7s". Any idea?

Code:
[root@scripts]# wget http://download.thinkbroadband.com/5MB.zip
--2013-04-21 01:32:47--  http://download.thinkbroadband.com/5MB.zip
Resolving download.thinkbroadband.com... 80.249.99.148, 2a02:68:1:7::1
Connecting to download.thinkbroadband.com|80.249.99.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5242880 (5.0M) [application/zip]
Saving to: `5MB.zip'

100%[==============================================================================================================================>] 5,242,880   1.34M/s   in 5.7s

2013-04-21 01:32:54 (900 KB/s) - `5MB.zip' saved [5242880/5242880]

---------- Post updated at 08:01 PM ---------- Previous update was at 07:38 PM ----------

Ah, figured it out. I used the last occurrence of "in" as the target as thats common across wget web page and file downloads.

Code:
#!/bin/bash
url_list=( http://www.bbc.co.uk http://www.cnn.com http://www.msn.com )

#Number of users to mimic simultaneously
users=20

#wget arguments
args="-E -H -T 30 -k -K -p --delete-after --no-cache -e robots=off"

function one_user () {
  local user=$1
  while [ 1 -eq 1 ]; do
    local wait=`expr $RANDOM % 120 + 30`
    local n=`expr $RANDOM % 3`
    local url=${url_list[$n]}
    time=`date +"%T"`
    date=`date +"%m-%d-%y"`
    wget=`wget $args $url 2>&1 | awk '/in/{a=$0}END{print a}' | awk -F "in" '{print$2}'`
    echo $date,$time,client$user,$url,$wget
    # echo user = $user wait = $wait url = $url
    sleep $wait
  done
  }

for (( user = 1; user <= $users; user++ )); do
  one_user $user &
done


Last edited by shadyuk; 04-20-2013 at 09:45 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

running shell script thru WEB page ....

....passing variable via list... here 's the HTML code extract : **************** <form method=post action=http://servername/cgi-bin/cgi-comptage_diff.ksh> <table border...........> .............. </table> <table bgcolor=#FFFFFF width="980"> ... (6 Replies)
Discussion started by: Nicol
6 Replies

2. Shell Programming and Scripting

Script to download file using wget

Hi I need a Shell script that will download a text file every second from a http server using wget. Can anyone provide me any pointers or sample scripts that will help me go about this task ??? regards techie (1 Reply)
Discussion started by: techie82
1 Replies

3. Shell Programming and Scripting

how to redirect to a web-page by shell script

Dear all, I am calling a korn shell script(CGI script) by a web-page. This shell script do some checking in a unix file and return true or false. Now within the same script, If it returns true then I want to redirect to another web-page stored in htdocs directory. Example: Login page sends a... (3 Replies)
Discussion started by: ravi18s
3 Replies

4. Shell Programming and Scripting

Perl script to copy contents of a web page

Hi All, Sorry to ask this question and i am not sure whether it is possible. please reply to my question. Thanks in advance. I need a perl script ( or any linux compatible scripts ) to copy the graphical contents of the webpage to a word pad. Say for example, i have a documentation site... (10 Replies)
Discussion started by: anand.linux1984
10 Replies

5. UNIX for Dummies Questions & Answers

Possible to download web page's text to a file?

Hi, Say there is a web page that contains just text only - that is, even the source code is just the text itself, nothing more. An example would be "http://mynasadata.larc.nasa.gov/docs/ocean_percent.txt" Is there a UNIX command that would allow me to download this text and store it in a... (1 Reply)
Discussion started by: Breanne
1 Replies

6. HP-UX

Help running a unix script from a web page

First, let me state that I am completely out of my realm with this. I have a server running HPUX. I'm not even sure if this can be considered a UNIX question and for that let me apologize in advance. I need to create a web page where a client can input 2 variables (i.e. date and phone number).... (0 Replies)
Discussion started by: grinds
0 Replies

7. UNIX for Dummies Questions & Answers

List and download web page files

Hello, Does anyone know of a way to list all files related to a single web page and then to download say 4 files at a time simultaneously until the entire web page has been downloaded successfully? I'm essentially trying to mimic a browser. Thanks. (2 Replies)
Discussion started by: shadyuk
2 Replies

8. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403... (2 Replies)
Discussion started by: Prasannag87
2 Replies

9. Shell Programming and Scripting

Refresh web page in bash script

hello, I am trying to refresh my web page which is created in bash script. I have a HTML page which when press a button calls a bash script. this bash script created the same page with dynamic data. When pressing the button I am calling to a function that set time out of 7 seconds and and after... (1 Reply)
Discussion started by: SH78
1 Replies

10. UNIX for Beginners Questions & Answers

How to use cURL to download web page with authentification (form)?

Hello, I'm new in the forum and really beginer, and also sorry form my bad english. I use linux and want to create little program to download automaticaly some pdf (invoices) and put in a folder of my computer. I learn how to do and programme with ubuntu but the program will be implemented... (1 Reply)
Discussion started by: MarcelOrMittal
1 Replies
APT-ZIP(8)																APT-ZIP(8)

NAME
apt-zip-list, apt-zip-inst - Use apt with removable media SYNOPSIS
apt-zip-list [ --help ] [ --version ] [ --medium=mountpoint ] [ --aptgetaction=action ] [ --packages=package[,package...] ] [ --fix-broken ] [ --skip-mount ] [ --method=method ] [ --options=opt[,opt...] ] [ --accept=accept[,accept...] | --reject=reject[,reject...] ] apt-zip-inst [ --help ] [ --version ] [ --medium=mountpoint ] [ --aptgetaction=action ] [ --packages=package[,package...] ] [ --fix-broken ] [ --skip-mount ] DESCRIPTION
This manpage should be up-to-date with version 0.17 of the programs. The apt-zip-list and apt-zip-inst commands simplify the upgrade process of a non-networked Debian host using apt(8), by using (preferably high-capacity) removable media, like a ZIP drive. apt-zip-list produces two files. One is a script that can be used on another host (maybe not running a Debian system) to fetch packages previously selected in dselect(8) or indicated in the command line, the other apt-zip.options saves the options used by apt-zip-list to indicate to apt-zip-inst what action to perform and/or which packages to install. Various types of scripts can be generated, by specifying a fetch method supported by the host that will fetch the packages. If you prefer, the script may generate a tarball containing the fetched packages. Note that apt-zip-list can be called as a non-root user. The name of the script (as well as the name of the tarfile when using the tar option) contains the name of the Debian host for which the packages are to be fetched. Thus you can, using the same medium, upgrade several hosts at the same time, provided you have enough space on this disk. This should work with or without the tar option. The script is written to the removable medium, which is mounted if necessary, and then restored to the original unmounted state when appli- cable. It is the responsibility of the user then to run the script on a host that will fetch the packages, and to have them (or the gener- ated archive) copied to the removable medium. When back on your Debian host, you just run apt-zip-inst, which will take care of the script-produced files, and call apt(8) to install the packages on your disk. OPTIONS
COMMON OPTIONS --help, -h Show summary of options. --version, -V Show version of program. --medium, -m Select a removable medium other than the site default. See also --skip-mount. --aptgetaction, -a Select the action done by apt-get. Possible actions are dselect-upgrade,upgrade and dist-upgrade. If --packages is not set the default is dselect-upgrade (See also apt-zip.conf), if it is set none is selected. --packages, -p Comma-separated list of extra packages to install. --fix-broken, -f Run apt-get with the --fix-broken option. See apt-get(8) for more details. --skip-mount, -s Allow specifying a non-mountpoint directory to the --medium option. This may help on laptops only getting point-to-point networking without being routed to. This is also useful while testing the program, as it allows sending the script into /tmp. --use-sleep=SEC, -S Introduces waiting times between downloaded files. Allows the genearation of a fetch script that will wait some time (config- urable). The number parameter represents the delay between consecutive dowloads. Note that this option is safe: even if the con- nected machine does not have a wait command, the script will not fail. --no-checksum, --no-md5, -5 This option will command to apt-zip-list to create a script which does not make any sha256sum checks. Note that if this option is not given at script generation time, the sha256sum command will be called only after it has been found in the system. (--no-md5 is maintained for script compatibility, but there is only sha256) OPTIONS FOR APT-ZIP-LIST --method, -M Select a method other than the default one. The wget and wget-dos methods are currently supported. --options, -o Specifies a set of options to be used while generating the script. A warning is issued only if an option is unkown to apt-zip-list, but all known options may not be supported by all methods. Options may be given a value, with the option=value syntax. Currently available options include: tar[=gnutarprogram] causes the dowloaded files to be grouped into a tarball. The name of a tar program on the download machine can be specified, if different from tar. Example: tar=gtar. When used in conjunction with the restart option, a GNU tar must be specified, so that 2 tarballs man be concatenated - this is only needed on the 3rd and subsequent runs of the fetch script. restart causes the downloaded files to be grouped into a tarball. --accept , -A / --reject , -R Comma-separated list of accepted/rejected protocols for download. By default are only accepted http and ftp. CONFIG FILE
The /etc/apt/apt-zip.conf file can be used to set defaults: method, removable medium, options, apt-get action and accepted/rejected proto- cols. It is a self-explainatory bourne-shell script snippet. It allows to provide site defaults for command-line options. METHOD API
A method is handled by an executable object (usually a script) placed in the /usr/share/apt-zip/methods/ directory. This executable is fed on standard input with the output of apt-get --print-uris. It is communicated the state of requested options using for each option an environment variables named OPTION_OPTNAME, where OPTNAME is the uppercased name of an option. Additional environment variables are also set, such as APTZIPVERSION, APTZIPTARFILE, APTZIPINSFILE, and OPTIONS. LIMITATIONS
o Only one disk can be used. If your packages do not fit on one, you have to deselect some packages using dselect(8) or you have to select a suitable list of package. o The files on an Internet distribution site can change rather frequently. It is up to the user to make sure the site was not updated between the build of the fetch script by apt-zip-list and the fetch itself, otherwise some required files may not be possible to fetch. FILES
/etc/apt/apt-zip.conf /usr/share/apt-zip/methods/ BUGS
o The restart feature is really slow. The tar command on Solaris(tm) has an option to seek in the file instead of assuming a non-seekable device, which greatly improves performance. GNU tar does not seem to be able to do that. o Virtually no error-checking is done. Examples of such error conditions include: - out-of-diskspace on both the removable medium and in apt archive cache when using tar option. o Somewhat consequently, error-recovery also nears non-existence. As examples, an interrupted fetch in tar mode has to be restarted from zero; changes in status file between runs of apt-zip-list and apt-zip-inst are not detected and may lead to failure. AUTHOR
The programs were written by YannDirson <dirson@debian.org>. It was maintained previously by SantiBejar <tiarda@yahoo.es>, and currently maintained by GiacomoCatenazzi <cate@debian.org> for the Debian GNU/Linux system. The source for this reference page is an SGML file, which can be converted to UNIX manpages using docbook2man(1) SEE ALSO
apt(8), apt-get(8), dselect(8). Debian GNU/Linux 01 February 2008 APT-ZIP(8)
All times are GMT -4. The time now is 10:01 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy