Help with perl script to search webpage

#1 (**permalink**) 11-28-2006

i have to point out that i'm a avid shell script lover and i have next to zero interest in perl which is why i dont know the first thing about it. however, just need to do this one thing on it..

i need to do something in perl but dont know how. I have a list of email addresses in the following format = firstname_lastname@company.com

now, i need to parse these emailaddresses to a webpage where i will search for the firstnames and lastnames of each user to see if they exist on the database of that webpage. its sorta like writing a perl script to automatically pull up the www.google.com page and search for first names and last names of any given people.

The search or request parameters are as follows,

Request parameters
search = phone

search text = first and last name

someone suggested i use the below but i have no clue how to do it.:

Post (HTTPRequest) first name and last name as using LWP to this url http://search.cpan.org/dist/libwww-perl/lib/LWP.pm

Please guys i would really appreciate your help in this.

#2 (**permalink**) 11-29-2006

Quote:

Originally Posted by Terrible

i have to point out that i'm a avid shell script lover and i have next to zero interest in perl which is why i dont know the first thing about it. however, just need to do this one thing on it..

i need to do something in perl but dont know how. I have a list of email addresses in the following format = firstname_lastname@company.com

now, i need to parse these emailaddresses to a webpage where i will search for the firstnames and lastnames of each user to see if they exist on the database of that webpage. its sorta like writing a perl script to automatically pull up the www.google.com page and search for first names and last names of any given people.

The search or request parameters are as follows,

Request parameters
search = phone

search text = first and last name

someone suggested i use the below but i have no clue how to do it.:

Post (HTTPRequest) first name and last name as using LWP to this url http://search.cpan.org/dist/libwww-perl/lib/LWP.pm

Please guys i would really appreciate your help in this.

any help with this?

cbkihong · #3 (**permalink**) 11-29-2006

Please rephrase your question, say with some illustration. I am not totally sure the exact question you have.

LWP is okay, but more complicated with its API. If you are just interested in simply access an HTTP URL and get the results, LWP::Simple is a simpler choice.

http://search.cpan.org/~gaas/libwww-.../LWP/Simple.pm

#4 (**permalink**) 11-30-2006

Quote:

Originally Posted by cbkihong

Please rephrase your question, say with some illustration. I am not totally sure the exact question you have.

LWP is okay, but more complicated with its API. If you are just interested in simply access an HTTP URL and get the results, LWP::Simple is a simpler choice.

http://search.cpan.org/~gaas/libwww-.../LWP/Simple.pm

ok. i have a list of names (first and last name) that i need to automatically search for on a site called, for instance, www.finestar.com.

on this site called finestar.com are search boxes where you can input the first name or last name of the person your looking for.

now, the list of names i have is quite enormous so manually searching for them on this site is virtually impossible or basically impractical.

enter perl

so i need to use perl to parse the list of names into the search boxes on the site called www.finestar.com and pull out any reference to the names being searched for.

cbkihong · #5 (**permalink**) 11-30-2006

Then, first of all you need to somehow reverse engineer the search form on the target site you are trying to connect to. Does it use POST or GET for the form submission? And investigate how the form is actually submitted, such as the parameters and the like, and whether any cookies may be required.

Actually for this, you can get a lot of help by using a packet sniffer to capture the HTTP request messages to the server. You simply use LWP to generate similar HTTP messages, and voila, just capture the response and parse it yourself. I suppose you know how to and what to parse yourself.

If you just read the documentation for LWP set of classes, you will see something similar to what I post below. In fact, the example below is nearly identical to one of the examples quoted in the manual:

Code:

#!/usr/bin/perl -w

use strict;

use LWP;

my $site = 'http://localhost/~bernardchan/test.php';

my $req = HTTP::Request->new(POST => $site);
$req->content_type('application/x-www-form-urlencoded');
$req->content('lastname=Green&firstname=John');

my $http = LWP::UserAgent->new();
my $res = $http->request($req);
if ($res->is_success) {
	my $body = $res->content;
	printf STDERR ("POST to %s successful\nContent goes below:\n", $site);
	print $body;
} else {
	printf STDERR ("ERROR: Cannot POST to %s (Code: %d)\n", $site, $res->code);
}

Code:

[bernardchan@allan shm]$ perl -Mstrict -w testlwp.pl 
POST to http://127.0.0.1/~bernardchan/test.php successful
Content goes below:
<html>
        <body>
                <h1>Search Results</h1>
                <p>This is a demo only.</p>
        </body>
</html>
[bernardchan@allan shm]$ perl -Mstrict -w testlwp.pl 
ERROR: Cannot POST to http://lfgjhdfkjhgldfjhgdfkg.com/~bernardchan/test.php (Code: 500)

Well, I think I need not clarify further, that mass POSTing for data mining is prohibited on most sites and you may be subject to bans by the system administrators should they find out. And it's just easy to find out, because everything is logged and I don't think any real sysadmin is dumb enough not to read the server logs.

#6 (**permalink**) 12-01-2006

Quote:

Originally Posted by cbkihong

Then, first of all you need to somehow reverse engineer the search form on the target site you are trying to connect to. Does it use POST or GET for the form submission? And investigate how the form is actually submitted, such as the parameters and the like, and whether any cookies may be required.

Actually for this, you can get a lot of help by using a packet sniffer to capture the HTTP request messages to the server. You simply use LWP to generate similar HTTP messages, and voila, just capture the response and parse it yourself. I suppose you know how to and what to parse yourself.

If you just read the documentation for LWP set of classes, you will see something similar to what I post below. In fact, the example below is nearly identical to one of the examples quoted in the manual:

Code:

#!/usr/bin/perl -w

use strict;

use LWP;

my $site = 'http://localhost/~bernardchan/test.php';

my $req = HTTP::Request->new(POST => $site);
$req->content_type('application/x-www-form-urlencoded');
$req->content('lastname=Green&firstname=John');

my $http = LWP::UserAgent->new();
my $res = $http->request($req);
if ($res->is_success) {
	my $body = $res->content;
	printf STDERR ("POST to %s successful\nContent goes below:\n", $site);
	print $body;
} else {
	printf STDERR ("ERROR: Cannot POST to %s (Code: %d)\n", $site, $res->code);
}

Code:

[bernardchan@allan shm]$ perl -Mstrict -w testlwp.pl 
POST to http://127.0.0.1/~bernardchan/test.php successful
Content goes below:
<html>
        <body>
                <h1>Search Results</h1>
                <p>This is a demo only.</p>
        </body>
</html>
[bernardchan@allan shm]$ perl -Mstrict -w testlwp.pl 
ERROR: Cannot POST to http://lfgjhdfkjhgldfjhgdfkg.com/~bernardchan/test.php (Code: 500)

Well, I think I need not clarify further, that mass POSTing for data mining is prohibited on most sites and you may be subject to bans by the system administrators should they find out. And it's just easy to find out, because everything is logged and I don't think any real sysadmin is dumb enough not to read the server logs.

thanks very much. i will give this a try. thanks a million.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Search and replace in Perl	jyoung	Shell Programming and Scripting	2	04-22-2008 02:05 PM
search & replace password perl script	shellscript22	Shell Programming and Scripting	4	03-25-2008 03:17 PM
search for a string -perl	meghana	Shell Programming and Scripting	11	02-12-2008 09:44 PM
Perl: Search for string on line then search and replace text	Crypto	Shell Programming and Scripting	4	01-04-2008 10:24 AM
Search string in array (perl)	man	Shell Programming and Scripting	6	07-24-2007 12:14 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by