Sponsored Content
Top Forums Shell Programming and Scripting How to get the page size (of a url) using wget Post 302100262 by Corona688 on Monday 18th of December 2006 09:52:48 AM
Old 12-18-2006
wget --spider --server-response [url] will print any headers the server returns without downloading the page proper. Unfortunately, many pages, particularly dynamically generated ones, won't report their size, just report "Length: unspecified [text/html]".
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

wget to check an URL

I all, I wrote an script which starts a Weblogic server and waits until its loaded to deploy several apps. The way I checked was something like: while ; do wget --spider <URL>:<port>/console > /dev/null 2>&1 rc=$? done This works perfectly because it's an HTML site and when server is... (2 Replies)
Discussion started by: AlbertGM
2 Replies

2. UNIX for Dummies Questions & Answers

wget with semicolon in page name

So, I'd like to wget a webpage, as its not going to stick around forever - but the problem is the webpage has a semicolon in it. wget http://example.com/stuff/asdf;asdf obviously doesn't get the right webpage. Any good way around this? (2 Replies)
Discussion started by: Julolidine
2 Replies

3. Shell Programming and Scripting

WGET cycling on an updating page

Hello, I am experiencing an issue while downloading a few pages using wget. All of them work without a problem except one which is a page that does a tail on the log and as a result is constantly getting updated. wget here seems to run endlessly and needs to be manually killed. I wanted to... (0 Replies)
Discussion started by: prafulnama
0 Replies

4. Shell Programming and Scripting

How to extract url from html page?

for example, I have an html file, contain <a href="http://awebsite" id="awebsite" class="first">website</a>and sometime a line contains more then one link, for example <a href="http://awebsite" id="awebsite" class="first">website</a><a href="http://bwebsite" id="bwebsite"... (36 Replies)
Discussion started by: 14th
36 Replies

5. UNIX for Dummies Questions & Answers

Launch a URL,validate username and password using wget or curl

Hi All, I want to launch "ex: http://gmail.com" from the cmd window and validate the credentials with username and password, is it possible? I have found something like this "wget --http-user=USER' --http-password=PASSWORD http://gmail.com" am new to this and unable to find a solution, i... (0 Replies)
Discussion started by: harsha85
0 Replies

6. Shell Programming and Scripting

Wget and single page

Good evening to all!! I'm trying to become familiar with wget. I would like to download a page from Wikipedia with all images and CSSs but without going down to all links present in the page. It should be named index.html. I would like also to save it to /mnt/us inside a new folder. This is... (5 Replies)
Discussion started by: silver18
5 Replies

7. UNIX for Dummies Questions & Answers

Read URL data from UNIX without wget,curl,lynx,w3m.

Hi Experts, Problem statement : We have an URL for which we need to read the data and get parsed inside the shell scripts. My Aix has very limited perl utility, i cant install any utility as well. Precisely, wget,cURL,Lynx,w3m and Lwp cant be used as i got these utilities only when i googled... (0 Replies)
Discussion started by: scott_cog
0 Replies

8. Shell Programming and Scripting

Read URL data from UNIX-CLI without Wget,CURL,w3m,LWP

Hi Experts, Problem statement : We have an URL for which we need to read the data and get parsed inside the shell scripts.My Aix has very limited perl utility, i cant install any utility as well. Precisely, wget,cURL,Lynx,w3m and Lwp cant be used as i got these utilities only when i googled it.... (12 Replies)
Discussion started by: scott_cog
12 Replies

9. Post Here to Contact Site Administrators and Moderators

Page Not Found error while parsing url

Hi I just tried to post following link while answering, its not parsing properly, just try on your browser Tried to paste while answering : https://www.unix.com/302873559-post2.htmlNot operator is not coming with HTML/PHP tags so attaching file (2 Replies)
Discussion started by: Akshay Hegde
2 Replies

10. Shell Programming and Scripting

Wget fails for a valid URL

Wget Error Codes: 0 No problems occurred. 1 Generic error code. 2 Parse error—for instance, when parsing command-line options, the .wgetrc or .netrc… 3 File I/O error. 4 Network failure. 5 SSL verification failure. 6 Username/password authentication failure. ... (3 Replies)
Discussion started by: mohtashims
3 Replies
WAPITI(1)							   User Commands							 WAPITI(1)

NAME
wapiti - a web application vulnerability scanner. SYNOPSIS
wapiti http://server.com/base/url/ [options] DESCRIPTION
Wapiti allows you to audit the security of your web applications. It performs "black-box" scans, i.e. it does not study the source code of the application but will scans the webpages of the deployed webapp, looking for scripts and forms where it can inject data. Once it gets this list, Wapiti acts like a fuzzer, injecting payloads to see if a script is vulnerable. OPTIONS
-s, --start <url> specify an url to start with. -x, --exclude <url> exclude an url from the scan (for example logout scripts) you can also use a wildcard (*): Example : -x "http://server/base/?page=*&module=test" or -x "http://server/base/admin/*" to exclude a directory -p, --proxy <url_proxy> specify a proxy (-p http://proxy:port/) -c, --cookie <cookie_file> use a cookie -t, --timeout <timeout> set the timeout (in seconds) -a, --auth <login%password> set credentials (for HTTP authentication) doesn't work with Python 2.4 -r, --remove <parameter_name> removes a parameter from URLs -m, --module <module> use a predefined set of scan/attack options: GET_ALL: only use GET request (no POST) GET_XSS: only XSS attacks with HTTP GET method POST_XSS: only XSS attacks with HTTP POST method -u, --underline use color to highlight vulnerable parameters in output -v, --verbose <level> set the verbosity level: 0: quiet (default), 1: print each url, 2: print every attack -h, --help print help page EFFICIENCY
Wapiti is developed in Python and use a library called lswww. This web spider library does the most of the work. Unfortunately, the html parsers module within python only works with well formed html pages so lswww fails to extract information from bad-coded webpages. Tidy can clean these webpages on the fly for us so lswww will give pretty good results. In order to make Wapiti far more efficient, you should: apt-get install python-utidylib python-ctypes AUTHOR
Copyright (C) 2006-2007 Nicolas Surribas <nicolas.surribas@gmail.com> Manpage created by Thomas Blasing <thomasbl@pool.math.tu-berlin.de> http://wapiti.sourceforge.net/ July 2007 WAPITI(1)
All times are GMT -4. The time now is 01:30 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy