Sponsored Content
Top Forums Shell Programming and Scripting Help with perl script to search webpage Post 302098250 by Terrible on Friday 1st of December 2006 01:23:44 PM
Old 12-01-2006
Quote:
Originally Posted by cbkihong
Then, first of all you need to somehow reverse engineer the search form on the target site you are trying to connect to. Does it use POST or GET for the form submission? And investigate how the form is actually submitted, such as the parameters and the like, and whether any cookies may be required.

Actually for this, you can get a lot of help by using a packet sniffer to capture the HTTP request messages to the server. You simply use LWP to generate similar HTTP messages, and voila, just capture the response and parse it yourself. I suppose you know how to and what to parse yourself.

If you just read the documentation for LWP set of classes, you will see something similar to what I post below. In fact, the example below is nearly identical to one of the examples quoted in the manual:

Code:
#!/usr/bin/perl -w

use strict;

use LWP;

my $site = 'http://localhost/~bernardchan/test.php';

my $req = HTTP::Request->new(POST => $site);
$req->content_type('application/x-www-form-urlencoded');
$req->content('lastname=Green&firstname=John');

my $http = LWP::UserAgent->new();
my $res = $http->request($req);
if ($res->is_success) {
	my $body = $res->content;
	printf STDERR ("POST to %s successful\nContent goes below:\n", $site);
	print $body;
} else {
	printf STDERR ("ERROR: Cannot POST to %s (Code: %d)\n", $site, $res->code);
}

Code:
[bernardchan@allan shm]$ perl -Mstrict -w testlwp.pl 
POST to http://127.0.0.1/~bernardchan/test.php successful
Content goes below:
<html>
        <body>
                <h1>Search Results</h1>
                <p>This is a demo only.</p>
        </body>
</html>
[bernardchan@allan shm]$ perl -Mstrict -w testlwp.pl 
ERROR: Cannot POST to http://lfgjhdfkjhgldfjhgdfkg.com/~bernardchan/test.php (Code: 500)

Well, I think I need not clarify further, that mass POSTing for data mining is prohibited on most sites and you may be subject to bans by the system administrators should they find out. And it's just easy to find out, because everything is logged and I don't think any real sysadmin is dumb enough not to read the server logs.

thanks very much. i will give this a try. thanks a million.
Terrible
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl script to search sprintf and replace with snprintf

Dear all, I am new to perl script and would need some help for my 1st script. I wrote a script to search sprintf(buf,"%s", sourcestring) and replace with snprintf(buf, sizeof(buf),"%s", sourcestring). As snprintf() requires an extra argument, so it is not a simple search-and-replace. I need to... (1 Reply)
Discussion started by: ChaMeN
1 Replies

2. Shell Programming and Scripting

Perl script to search and extract using wildcards.

Good evening All, I have a perl script to pull out all occurrences of a files beginning with xx and ending in .p. I will then loop through all 1K files in a directory. I can grep for xx*.p files but it gives me the entire line. I wish to output to a single colum with only the hits found. ... (3 Replies)
Discussion started by: CammyD
3 Replies

3. Shell Programming and Scripting

Code help with search script perl

Hello, I have a script, but its sorta broke and I need help to tweak it #!/usr/bin/perl my @files_to_search = ("exportOBJ-sark.txt","ResourceRecText-ALL.txt","exportdnsrr-report.txt"); open (FL, "sark.com.txt"); my %search_hash = (); while (my $line = <FL>) { my @parts =... (12 Replies)
Discussion started by: richsark
12 Replies

4. Shell Programming and Scripting

perl script to search n place a pattern

hi, i have one file as t1.txt as below hi hello welcome i want perl script to search for the pattern "abcd" in the file. if the pattern doesn't exist, i want to insert that pattern at the end of the same file t1.txt my o/p should be hi hllo welcome abcd thank you (3 Replies)
Discussion started by: roopa
3 Replies

5. Shell Programming and Scripting

open a webpage through perl script

Hi All Can anyone please help me how to open a webpage suppose(Google) with the help of perl script and refresh it after every 5 minutes. (0 Replies)
Discussion started by: parthmittal2007
0 Replies

6. Shell Programming and Scripting

Navigating a WebPage with Perl

Hi All Below is Code, It opens a link from which it ask a login name and password, the script enter the login name and password and navigate to next page.. In the next page there is a drop down box from which i have to select a value, I have written the code but it gives error #!/usr/bin/perl... (3 Replies)
Discussion started by: parthmittal2007
3 Replies

7. Shell Programming and Scripting

parsing a webpage - perl lwp

I am requesting for the text parsing section below. Any helps are highly appreciated. <tr valign="top"><td nowrap>Source name</td> <td style="text-align: justify">Sample Name<br></td> I want Sample Name from above. In the same file, I have to search for another pattern like this <td><a... (1 Reply)
Discussion started by: jacobs.smith
1 Replies

8. Shell Programming and Scripting

How to select a particular field from a drop-down menu in a webpage using perl script?

Hi Team, I have a requirement to login into URL using username and password , then I have to select a "particular name" from drop-down menu and then Read the values user records etc.... using perl. Is it possible to do in perl script ? (OR) Can you please let me know which scripting... (1 Reply)
Discussion started by: latika
1 Replies

9. Shell Programming and Scripting

Perl Script - Sort Columns on webpage

foreach my $ds (sort { $a->{books} cmp $b->{books} || (($a->{books} eq "A") ? ($a->{hammers} cmp $b->{hammers}) : (($a->{shoes} <=> $b->{shoes}) || ($a->{hammers}... (1 Reply)
Discussion started by: scj2012
1 Replies

10. Shell Programming and Scripting

Applescript to open webpage from list, search for "text"

hi guys, its me, doritos guy, I've laid off of the doritos for a while, twas given me stomach problems. you know when you eat 10 family size bags a day it might cause a problem, any-who i appreciated the help from last time when i was trying to script something, now i just have to try to... (2 Replies)
Discussion started by: ilovedoritos
2 Replies
Mojo::Message::Request(3pm)				User Contributed Perl Documentation			       Mojo::Message::Request(3pm)

NAME
Mojo::Message::Request - HTTP 1.1 request container SYNOPSIS
use Mojo::Message::Request; # Parse my $req = Mojo::Message::Request->new; $req->parse("GET /foo HTTP/1.0x0ax0d"); $req->parse("Content-Length: 12x0ax0dx0ax0d"); $req->parse("Content-Type: text/plainx0ax0dx0ax0d"); $req->parse('Hello World!'); say $req->body; # Build my $req = Mojo::Message::Request->new; $req->url->parse('http://127.0.0.1/foo/bar'); $req->method('GET'); say $req->to_string; DESCRIPTION
Mojo::Message::Request is a container for HTTP 1.1 requests as described in RFC 2616. EVENTS
Mojo::Message::Request inherits all events from Mojo::Message. ATTRIBUTES
Mojo::Message::Request inherits all attributes from Mojo::Message and implements the following new ones. "env" my $env = $req->env; $req = $req->env({}); Direct access to the "CGI" or "PSGI" environment hash if available. # Check CGI version my $version = $req->env->{GATEWAY_INTERFACE}; # Check PSGI version my $version = $req->env->{'psgi.version'}; "method" my $method = $req->method; $req = $req->method('POST'); HTTP request method, defaults to "GET". "url" my $url = $req->url; $req = $req->url(Mojo::URL->new); HTTP request URL, defaults to a Mojo::URL object. my $foo = $req->url->query->to_hash->{foo}; METHODS
Mojo::Message::Request inherits all methods from Mojo::Message and implements the following new ones. "clone" my $clone = $req->clone; Clone request if possible, otherwise return "undef". "cookies" my $cookies = $req->cookies; $req = $req->cookies(Mojo::Cookie::Request->new); $req = $req->cookies({name => 'foo', value => 'bar'}); Access request cookies, usually Mojo::Cookie::Request objects. say $req->cookies->[1]->value; "fix_headers" $req = $req->fix_headers; Make sure request has all required headers for the current HTTP version. "is_secure" my $success = $req->is_secure; Check if connection is secure. "is_xhr" my $success = $req->is_xhr; Check "X-Requested-With" header for "XMLHttpRequest" value. "param" my @names = $req->param; my $foo = $req->param('foo'); my @foo = $req->param('foo'); Access "GET" and "POST" parameters. Note that this method caches all data, so it should not be called before the entire request body has been received. "params" my $p = $req->params; All "GET" and "POST" parameters, usually a Mojo::Parameters object. say $req->params->param('foo'); "parse" $req = $req->parse('GET /foo/bar HTTP/1.1'); $req = $req->parse(REQUEST_METHOD => 'GET'); $req = $req->parse({REQUEST_METHOD => 'GET'}); Parse HTTP request chunks or environment hash. "proxy" my $proxy = $req->proxy; $req = $req->proxy('http://foo:bar@127.0.0.1:3000'); $req = $req->proxy(Mojo::URL->new('http://127.0.0.1:3000')); Proxy URL for request. # Disable proxy $req->proxy(0); "query_params" my $p = $req->query_params; All "GET" parameters, usually a Mojo::Parameters object. say $req->query_params->to_hash->{'foo'}; SEE ALSO
Mojolicious, Mojolicious::Guides, <http://mojolicio.us>. perl v5.14.2 2012-09-05 Mojo::Message::Request(3pm)
All times are GMT -4. The time now is 10:04 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy