Newbie Python Url Scraper Post: 302816785

Sponsored Content

Top Forums Shell Programming and Scripting Newbie Python Url Scraper Post 302816785 by metallica1973 on Tuesday 4th of June 2013 02:40:02 PM

06-04-2013

Registered User

Thanks for the reply. It didnt make a difference.

Code:

wget --spider r --server-response -U netscape http://192.168.3.3

Spider mode enabled. Check if remote file exists.
--2013-06-04 14:34:51--  (try:19)  http://192.168.3.3/
Connecting to 192.168.3.3:80... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

Spider mode enabled. Check if remote file exists.
--2013-06-04 14:35:01--  (try:20)  http://192.168.3.3/
Connecting to 192.168.3.3:80... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Giving up.

I tried just a basic HTTP GET request and got this:

Code:

nc -v 192.168.3.3 80
Connection to 192.168.3.3 80 port [tcp/http] succeeded!
GET / HTTP/1.0

HTTP/1.1 400 Bad Request
Server: Netwave IP Camera
Date: Tue, 04 Jun 2013 18:37:28 GMT
Content-Type: text/html
Content-Length: 135
Connection: close

<HTML><HEAD><TITLE>400 Bad Request</TITLE></HEAD>
<BODY BGCOLOR="#cc9999"><H4>400 Bad Request</H4>
Can't parse request.
</BODY></HTML>

I will dig around. Thanks

metallica1973

View Public Profile for metallica1973

Find all posts by metallica1973

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss

2. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss

3. Programming

NEWBIE QUESTION: python 3 or 2.6.x

I'm a newbie and want to learn a programming language, willy-nilly I picked python... Should I go with 2.6.x which at first glance seems extremely well documented, or should I go with 3.0, which is new and shiny?! I want...no...I'm going to NEED fantastic documentation or I'm going to fail...

4. UNIX for Dummies Questions & Answers

ReDirecting a URL to another URL - Linux

Hello, I need to redirect an existing URL, how can i do that? There's a current web address to a GUI that I have to redirect to another webaddress. Does anyone know how to do this? This is on Unix boxes Linux. example: https://m45.testing.address.net/host.php make it so the...

5. UNIX for Dummies Questions & Answers

UNIX newbie NEWBIE question!

Hello everyone, Just started UNIX today! In our school we use solaris. I just want to know how do I setup Solaris 10 not the GUI one, the one where you have to type the commands like ECHO, ls, pwd, etc... I have windows xp and I also have vmware. I hope I am not missing anything! :p

6. Web Development

Regex to rewrite URL to another URL based on HTTP_HOST?

I am trying to find a way to test some code, but I need to rewrite a specific URL only from a specific HTTP_HOST The call goes out to http://SUB.DOMAIN.COM/showAssignment/7bde10b45efdd7a97629ef2fe01f7303/jsmodule/Nevow.Athena The ID in the middle is always random due to the cookie. I...

7. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1...

8. Shell Programming and Scripting

Python Newbie Question Regex

I starting teaching myself python and am stuck on trying to understand why I am not getting the output that I want. Long story short, I am using PDB for debugging and here my function in which I am having my issue: import re ... ... ... def find_all_flvs(url): soup =...

9. Shell Programming and Scripting

Reading URL using Mechanize and dump all the contents of the URL to a file

Hello, Am very new to perl , please help me here !! I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file. below is the script which i have written so far , #!/usr/bin/perl use LWP::UserAgent; use...

LEARN ABOUT OPENSOLARIS

prtdscp

prtdscp(1M)						  System Administration Commands					       prtdscp(1M)

NAME

       prtdscp - display DSCP IP addresses

SYNOPSIS

       prtdscp [-v ]

       prtdscp [-v ] -h

       prtdscp [-v ] -d

       prtdscp [-v ] -s

DESCRIPTION

       prtdscp	displays  the  IP addresses associated with a Domain to Service Processor Communications Protocol (DSCP) link. If no arguments are
       specified, prtdscp displays the IP addresses on both ends of the DSCP link. The IP address of either the Service Processor or  domain  side
       can be displayed separately by the use of the -s or -d options, respectively.

OPTIONS

       The following options are supported:

       -v    Verbose mode. Print additional details about the program's internal progress to stderr.

       -h    Help. Print a brief synopsis of the program's usage and exit. All other command line arguments are ignored.

       -d    Display only the local domain's IP address.

       -s    Display only the remote Service Processor's IP address.

EXAMPLES

       Example 1 Displaying both addresses

       The following example displays both the local domain's IP address and the remote SP's IP address:

	 # prtdscp
	 Domain Address: 192.168.103.2
	 SP Address: 192.168.103.1

       Example 2 Displaying the local IP address

       The following example displays the local domain's IP address:

	 # prtdscp -d
	 192.168.103.2

       Example 3 Displaying the remote IP address

       The following example display the remote SP's IP address:

	 # prtdscp -s
	 192.168.103.1

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWdscpr.u, SUNWdscpu.u	   |
       +-----------------------------+-----------------------------+
       |Interface Stability	     |Evolving			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       attributes(5)

SunOS 5.11							    25 Apr 2006 						       prtdscp(1M)

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Discussion started by: gander_ss

2. Shell Programming and Scripting

url calling and parameter passing to url in script

Discussion started by: gander_ss

3. Programming

NEWBIE QUESTION: python 3 or 2.6.x

Discussion started by: guptaxpn

4. UNIX for Dummies Questions & Answers

ReDirecting a URL to another URL - Linux

Discussion started by: SkySmart

5. UNIX for Dummies Questions & Answers

UNIX newbie NEWBIE question!

Discussion started by: Hanamachi

6. Web Development

Regex to rewrite URL to another URL based on HTTP_HOST?

Discussion started by: EXT3FSCK

7. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Discussion started by: striker4o

8. Shell Programming and Scripting

Python Newbie Question Regex

Discussion started by: metallica1973

9. Shell Programming and Scripting

Reading URL using Mechanize and dump all the contents of the URL to a file

Discussion started by: scott_cog

LEARN ABOUT OPENSOLARIS

prtdscp