Then, first of all you need to somehow reverse engineer the search form on the target site you are trying to connect to. Does it use POST or GET for the form submission? And investigate how the form is actually submitted, such as the parameters and the like, and whether any cookies may be required.
Actually for this, you can get a lot of help by using a packet sniffer to capture the HTTP request messages to the server. You simply use LWP to generate similar HTTP messages, and voila, just capture the response and parse it yourself. I suppose you know how to and what to parse yourself.
If you just read the documentation for LWP set of classes, you will see something similar to what I post below. In fact, the example below is nearly identical to one of the examples quoted in the manual:
Well, I think I need not clarify further, that mass POSTing for data mining is prohibited on most sites and you may be subject to bans by the system administrators should they find out. And it's just easy to find out, because everything is logged and I don't think any real sysadmin is dumb enough not to read the server logs.
thanks very much. i will give this a try. thanks a million.
Dear all,
I am new to perl script and would need some help for my 1st script. I wrote a script to search sprintf(buf,"%s", sourcestring) and replace with snprintf(buf, sizeof(buf),"%s", sourcestring). As snprintf() requires an extra argument, so it is not a simple search-and-replace. I need to... (1 Reply)
Good evening All,
I have a perl script to pull out all occurrences of a files beginning with xx and ending in .p. I will then loop through all 1K files in a directory. I can grep for xx*.p files but it gives me the entire line. I wish to output to a single colum with only the hits found. ... (3 Replies)
Hello,
I have a script, but its sorta broke and I need help to tweak it
#!/usr/bin/perl
my @files_to_search = ("exportOBJ-sark.txt","ResourceRecText-ALL.txt","exportdnsrr-report.txt");
open (FL, "sark.com.txt");
my %search_hash = ();
while (my $line = <FL>) {
my @parts =... (12 Replies)
hi,
i have one file as t1.txt as below
hi
hello
welcome
i want perl script to search for the pattern "abcd" in the file.
if the pattern doesn't exist, i want to insert that pattern at the end of the same file t1.txt
my o/p should be
hi
hllo
welcome
abcd
thank you (3 Replies)
Hi All
Below is Code, It opens a link from which it ask a login name and password, the script enter the login name and password and navigate to next page..
In the next page there is a drop down box from which i have to select a value, I have written the code but it gives error
#!/usr/bin/perl... (3 Replies)
I am requesting for the text parsing section below. Any helps are highly appreciated.
<tr valign="top"><td nowrap>Source name</td>
<td style="text-align: justify">Sample Name<br></td>
I want Sample Name from above.
In the same file, I have to search for another pattern like this
<td><a... (1 Reply)
Hi Team,
I have a requirement to login into URL using username and password , then I have to select a "particular name" from drop-down menu and then Read the values user records etc.... using perl.
Is it possible to do in perl script ? (OR) Can you please let me know which scripting... (1 Reply)
hi guys, its me, doritos guy, I've laid off of the doritos for a while, twas given me stomach problems. you know when you eat 10 family size bags a day it might cause a problem, any-who
i appreciated the help from last time when i was trying to script something, now i just have to try to... (2 Replies)
Discussion started by: ilovedoritos
2 Replies
LEARN ABOUT DEBIAN
mojo::message::request
Mojo::Message::Request(3pm) User Contributed Perl Documentation Mojo::Message::Request(3pm)NAME
Mojo::Message::Request - HTTP 1.1 request container
SYNOPSIS
use Mojo::Message::Request;
# Parse
my $req = Mojo::Message::Request->new;
$req->parse("GET /foo HTTP/1.0x0ax0d");
$req->parse("Content-Length: 12x0ax0dx0ax0d");
$req->parse("Content-Type: text/plainx0ax0dx0ax0d");
$req->parse('Hello World!');
say $req->body;
# Build
my $req = Mojo::Message::Request->new;
$req->url->parse('http://127.0.0.1/foo/bar');
$req->method('GET');
say $req->to_string;
DESCRIPTION
Mojo::Message::Request is a container for HTTP 1.1 requests as described in RFC 2616.
EVENTS
Mojo::Message::Request inherits all events from Mojo::Message.
ATTRIBUTES
Mojo::Message::Request inherits all attributes from Mojo::Message and implements the following new ones.
"env"
my $env = $req->env;
$req = $req->env({});
Direct access to the "CGI" or "PSGI" environment hash if available.
# Check CGI version
my $version = $req->env->{GATEWAY_INTERFACE};
# Check PSGI version
my $version = $req->env->{'psgi.version'};
"method"
my $method = $req->method;
$req = $req->method('POST');
HTTP request method, defaults to "GET".
"url"
my $url = $req->url;
$req = $req->url(Mojo::URL->new);
HTTP request URL, defaults to a Mojo::URL object.
my $foo = $req->url->query->to_hash->{foo};
METHODS
Mojo::Message::Request inherits all methods from Mojo::Message and implements the following new ones.
"clone"
my $clone = $req->clone;
Clone request if possible, otherwise return "undef".
"cookies"
my $cookies = $req->cookies;
$req = $req->cookies(Mojo::Cookie::Request->new);
$req = $req->cookies({name => 'foo', value => 'bar'});
Access request cookies, usually Mojo::Cookie::Request objects.
say $req->cookies->[1]->value;
"fix_headers"
$req = $req->fix_headers;
Make sure request has all required headers for the current HTTP version.
"is_secure"
my $success = $req->is_secure;
Check if connection is secure.
"is_xhr"
my $success = $req->is_xhr;
Check "X-Requested-With" header for "XMLHttpRequest" value.
"param"
my @names = $req->param;
my $foo = $req->param('foo');
my @foo = $req->param('foo');
Access "GET" and "POST" parameters. Note that this method caches all data, so it should not be called before the entire request body has
been received.
"params"
my $p = $req->params;
All "GET" and "POST" parameters, usually a Mojo::Parameters object.
say $req->params->param('foo');
"parse"
$req = $req->parse('GET /foo/bar HTTP/1.1');
$req = $req->parse(REQUEST_METHOD => 'GET');
$req = $req->parse({REQUEST_METHOD => 'GET'});
Parse HTTP request chunks or environment hash.
"proxy"
my $proxy = $req->proxy;
$req = $req->proxy('http://foo:bar@127.0.0.1:3000');
$req = $req->proxy(Mojo::URL->new('http://127.0.0.1:3000'));
Proxy URL for request.
# Disable proxy
$req->proxy(0);
"query_params"
my $p = $req->query_params;
All "GET" parameters, usually a Mojo::Parameters object.
say $req->query_params->to_hash->{'foo'};
SEE ALSO
Mojolicious, Mojolicious::Guides, <http://mojolicio.us>.
perl v5.14.2 2012-09-05 Mojo::Message::Request(3pm)