Help with perl script to search webpage


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with perl script to search webpage
# 1  
Old 11-28-2006
Help with perl script to search webpage

i have to point out that i'm a avid shell script lover and i have next to zero interest in perl which is why i dont know the first thing about it. however, just need to do this one thing on it..

i need to do something in perl but dont know how. I have a list of email addresses in the following format = firstname_lastname@company.com

now, i need to parse these emailaddresses to a webpage where i will search for the firstnames and lastnames of each user to see if they exist on the database of that webpage. its sorta like writing a perl script to automatically pull up the www.google.com page and search for first names and last names of any given people.


The search or request parameters are as follows,

Request parameters
search = phone

search text = first and last name



someone suggested i use the below but i have no clue how to do it.:

Post (HTTPRequest) first name and last name as using LWP to this url http://search.cpan.org/dist/libwww-perl/lib/LWP.pm


Please guys i would really appreciate your help in this.
Terrible
# 2  
Old 11-29-2006
Quote:
Originally Posted by Terrible
i have to point out that i'm a avid shell script lover and i have next to zero interest in perl which is why i dont know the first thing about it. however, just need to do this one thing on it..

i need to do something in perl but dont know how. I have a list of email addresses in the following format = firstname_lastname@company.com

now, i need to parse these emailaddresses to a webpage where i will search for the firstnames and lastnames of each user to see if they exist on the database of that webpage. its sorta like writing a perl script to automatically pull up the www.google.com page and search for first names and last names of any given people.


The search or request parameters are as follows,

Request parameters
search = phone

search text = first and last name



someone suggested i use the below but i have no clue how to do it.:

Post (HTTPRequest) first name and last name as using LWP to this url http://search.cpan.org/dist/libwww-perl/lib/LWP.pm


Please guys i would really appreciate your help in this.

any help with this?
Terrible
# 3  
Old 11-29-2006
Please rephrase your question, say with some illustration. I am not totally sure the exact question you have.

LWP is okay, but more complicated with its API. If you are just interested in simply access an HTTP URL and get the results, LWP::Simple is a simpler choice.

http://search.cpan.org/~gaas/libwww-.../LWP/Simple.pm
# 4  
Old 11-30-2006
Quote:
Originally Posted by cbkihong
Please rephrase your question, say with some illustration. I am not totally sure the exact question you have.

LWP is okay, but more complicated with its API. If you are just interested in simply access an HTTP URL and get the results, LWP::Simple is a simpler choice.

http://search.cpan.org/~gaas/libwww-.../LWP/Simple.pm

ok. i have a list of names (first and last name) that i need to automatically search for on a site called, for instance, www.finestar.com.

on this site called finestar.com are search boxes where you can input the first name or last name of the person your looking for.

now, the list of names i have is quite enormous so manually searching for them on this site is virtually impossible or basically impractical.

enter perl

so i need to use perl to parse the list of names into the search boxes on the site called www.finestar.com and pull out any reference to the names being searched for.
Terrible
# 5  
Old 11-30-2006
Then, first of all you need to somehow reverse engineer the search form on the target site you are trying to connect to. Does it use POST or GET for the form submission? And investigate how the form is actually submitted, such as the parameters and the like, and whether any cookies may be required.

Actually for this, you can get a lot of help by using a packet sniffer to capture the HTTP request messages to the server. You simply use LWP to generate similar HTTP messages, and voila, just capture the response and parse it yourself. I suppose you know how to and what to parse yourself.

If you just read the documentation for LWP set of classes, you will see something similar to what I post below. In fact, the example below is nearly identical to one of the examples quoted in the manual:

Code:
#!/usr/bin/perl -w

use strict;

use LWP;

my $site = 'http://localhost/~bernardchan/test.php';

my $req = HTTP::Request->new(POST => $site);
$req->content_type('application/x-www-form-urlencoded');
$req->content('lastname=Green&firstname=John');

my $http = LWP::UserAgent->new();
my $res = $http->request($req);
if ($res->is_success) {
	my $body = $res->content;
	printf STDERR ("POST to %s successful\nContent goes below:\n", $site);
	print $body;
} else {
	printf STDERR ("ERROR: Cannot POST to %s (Code: %d)\n", $site, $res->code);
}

Code:
[bernardchan@allan shm]$ perl -Mstrict -w testlwp.pl 
POST to http://127.0.0.1/~bernardchan/test.php successful
Content goes below:
<html>
        <body>
                <h1>Search Results</h1>
                <p>This is a demo only.</p>
        </body>
</html>
[bernardchan@allan shm]$ perl -Mstrict -w testlwp.pl 
ERROR: Cannot POST to http://lfgjhdfkjhgldfjhgdfkg.com/~bernardchan/test.php (Code: 500)

Well, I think I need not clarify further, that mass POSTing for data mining is prohibited on most sites and you may be subject to bans by the system administrators should they find out. And it's just easy to find out, because everything is logged and I don't think any real sysadmin is dumb enough not to read the server logs.
# 6  
Old 12-01-2006
Quote:
Originally Posted by cbkihong
Then, first of all you need to somehow reverse engineer the search form on the target site you are trying to connect to. Does it use POST or GET for the form submission? And investigate how the form is actually submitted, such as the parameters and the like, and whether any cookies may be required.

Actually for this, you can get a lot of help by using a packet sniffer to capture the HTTP request messages to the server. You simply use LWP to generate similar HTTP messages, and voila, just capture the response and parse it yourself. I suppose you know how to and what to parse yourself.

If you just read the documentation for LWP set of classes, you will see something similar to what I post below. In fact, the example below is nearly identical to one of the examples quoted in the manual:

Code:
#!/usr/bin/perl -w

use strict;

use LWP;

my $site = 'http://localhost/~bernardchan/test.php';

my $req = HTTP::Request->new(POST => $site);
$req->content_type('application/x-www-form-urlencoded');
$req->content('lastname=Green&firstname=John');

my $http = LWP::UserAgent->new();
my $res = $http->request($req);
if ($res->is_success) {
	my $body = $res->content;
	printf STDERR ("POST to %s successful\nContent goes below:\n", $site);
	print $body;
} else {
	printf STDERR ("ERROR: Cannot POST to %s (Code: %d)\n", $site, $res->code);
}

Code:
[bernardchan@allan shm]$ perl -Mstrict -w testlwp.pl 
POST to http://127.0.0.1/~bernardchan/test.php successful
Content goes below:
<html>
        <body>
                <h1>Search Results</h1>
                <p>This is a demo only.</p>
        </body>
</html>
[bernardchan@allan shm]$ perl -Mstrict -w testlwp.pl 
ERROR: Cannot POST to http://lfgjhdfkjhgldfjhgdfkg.com/~bernardchan/test.php (Code: 500)

Well, I think I need not clarify further, that mass POSTing for data mining is prohibited on most sites and you may be subject to bans by the system administrators should they find out. And it's just easy to find out, because everything is logged and I don't think any real sysadmin is dumb enough not to read the server logs.

thanks very much. i will give this a try. thanks a million.
Terrible
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Applescript to open webpage from list, search for "text"

hi guys, its me, doritos guy, I've laid off of the doritos for a while, twas given me stomach problems. you know when you eat 10 family size bags a day it might cause a problem, any-who i appreciated the help from last time when i was trying to script something, now i just have to try to... (2 Replies)
Discussion started by: ilovedoritos
2 Replies

2. Shell Programming and Scripting

Perl Script - Sort Columns on webpage

foreach my $ds (sort { $a->{books} cmp $b->{books} || (($a->{books} eq "A") ? ($a->{hammers} cmp $b->{hammers}) : (($a->{shoes} <=> $b->{shoes}) || ($a->{hammers}... (1 Reply)
Discussion started by: scj2012
1 Replies

3. Shell Programming and Scripting

How to select a particular field from a drop-down menu in a webpage using perl script?

Hi Team, I have a requirement to login into URL using username and password , then I have to select a "particular name" from drop-down menu and then Read the values user records etc.... using perl. Is it possible to do in perl script ? (OR) Can you please let me know which scripting... (1 Reply)
Discussion started by: latika
1 Replies

4. Shell Programming and Scripting

parsing a webpage - perl lwp

I am requesting for the text parsing section below. Any helps are highly appreciated. <tr valign="top"><td nowrap>Source name</td> <td style="text-align: justify">Sample Name<br></td> I want Sample Name from above. In the same file, I have to search for another pattern like this <td><a... (1 Reply)
Discussion started by: jacobs.smith
1 Replies

5. Shell Programming and Scripting

Navigating a WebPage with Perl

Hi All Below is Code, It opens a link from which it ask a login name and password, the script enter the login name and password and navigate to next page.. In the next page there is a drop down box from which i have to select a value, I have written the code but it gives error #!/usr/bin/perl... (3 Replies)
Discussion started by: parthmittal2007
3 Replies

6. Shell Programming and Scripting

open a webpage through perl script

Hi All Can anyone please help me how to open a webpage suppose(Google) with the help of perl script and refresh it after every 5 minutes. (0 Replies)
Discussion started by: parthmittal2007
0 Replies

7. Shell Programming and Scripting

perl script to search n place a pattern

hi, i have one file as t1.txt as below hi hello welcome i want perl script to search for the pattern "abcd" in the file. if the pattern doesn't exist, i want to insert that pattern at the end of the same file t1.txt my o/p should be hi hllo welcome abcd thank you (3 Replies)
Discussion started by: roopa
3 Replies

8. Shell Programming and Scripting

Code help with search script perl

Hello, I have a script, but its sorta broke and I need help to tweak it #!/usr/bin/perl my @files_to_search = ("exportOBJ-sark.txt","ResourceRecText-ALL.txt","exportdnsrr-report.txt"); open (FL, "sark.com.txt"); my %search_hash = (); while (my $line = <FL>) { my @parts =... (12 Replies)
Discussion started by: richsark
12 Replies

9. Shell Programming and Scripting

Perl script to search and extract using wildcards.

Good evening All, I have a perl script to pull out all occurrences of a files beginning with xx and ending in .p. I will then loop through all 1K files in a directory. I can grep for xx*.p files but it gives me the entire line. I wish to output to a single colum with only the hits found. ... (3 Replies)
Discussion started by: CammyD
3 Replies

10. Shell Programming and Scripting

Perl script to search sprintf and replace with snprintf

Dear all, I am new to perl script and would need some help for my 1st script. I wrote a script to search sprintf(buf,"%s", sourcestring) and replace with snprintf(buf, sizeof(buf),"%s", sourcestring). As snprintf() requires an extra argument, so it is not a simple search-and-replace. I need to... (1 Reply)
Discussion started by: ChaMeN
1 Replies
Login or Register to Ask a Question