Visit Our UNIX and Linux User Community


Reading URL using Mechanize and dump all the contents of the URL to a file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Reading URL using Mechanize and dump all the contents of the URL to a file
# 1  
Old 02-12-2014
Reading URL using Mechanize and dump all the contents of the URL to a file

Hello,
Am very new to perl , please help me here !!

I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file.

below is the script which i have written so far ,

Code:
#!/usr/bin/perl
use LWP::UserAgent;
use WWW::Mechanize;
use WWW::DecodedContent;
use LWP::Debug qw( + );
use HTTP::Headers ;
my $mech = WWW::Mechanize->new( autocheck => 1 );
$mech->status();
print "Am inside Mechanize\n" ;

my $content = $mech->decoded_content || $mech->content;
#$mech->response()->decoded_content();
$ua->default_header("Accept-Encoding" => "gzip, deflate");
#$mech = WWW::Mechanize->new ;
$ua->agent("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
#$ua->default_header("Accept-Encoding" => "gzip, deflate");
$mech->get( "http://zzzzzzzzzURLzzzzzzzz" );
my $test = $mech->forms();
die "Can't even get the home page: ", $mech->response->status_line
unless $mech->success;

$mech->status();
$mech->uri();
 if ($response->is_success) {
   print "\n i am in finally";
   }
   else {
          die $response->status_line;
 }

Please feel free to provide some suggestion on changing the original script ,this is my first draft of code.

where as am getting an error " Can't call method "default_header" on an undefined value at getjan23.pl line 14. "

please let me know what am missing.

Thanks in advance

Last edited by scott_cog; 02-12-2014 at 11:05 AM.. Reason: .
# 2  
Old 02-14-2014
Check the wget utility, i guess it does what you need ... without having to reinvent the wheel Smilie
# 3  
Old 02-17-2014
Hi Ctsgnb,

Thanks for your reply.

Actually in the server we dont have wget ,curl utility and we cannot install as well.

Btw, i found the answer and i was able to get the output which i supposed to get. Hope it will be useful for others who are in search of such thing. and it was pretty simple.

Code:
#!/usr/bin/perl

use strict;
use warnings;
use LWP::UserAgent;

my $req;
my $res;

my $ua = LWP::UserAgent->new;
$ua->agent("Firefox/1.5.0.10");
$ua->timeout(1000);

$req = HTTP::Request->new(GET => 'http://yyyzzzzzxxxxxxxiiiii');

$res = $ua->request($req);

if ($res->is_success) {
    print $res->content;
}
else {
    print "Error: " . $res->status . "\n";
}


Last edited by scott_cog; 02-17-2014 at 11:22 AM.. Reason: .

Previous Thread | Next Thread
Test Your Knowledge in Computers #876
Difficulty: Medium
All operational code must be POSIX compliant in order to be cost effective.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extracting the file name from the specified URL

Hello Everyone, I am trying to write a shell script(or Perl Script) that would do the following: I have a file that contains the following lines: File: https://ims-svnus.com/dev/DB/trunk/feeds/templates/shell_script.txt -r860... (5 Replies)
Discussion started by: filter
5 Replies

2. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

3. Shell Programming and Scripting

Given a url how to get last part of file

HI, I have a URL that points to a file: LINK= "http://www.webpage.org/project/team2/file.tar" However when I try to use wget on this variable I receive the following error. wget $LINK line 4: http://www.webpage.org/project/team2/file.tar: No such file or directory wget:... (1 Reply)
Discussion started by: bashnewbee
1 Replies

4. Web Development

Regex to rewrite URL to another URL based on HTTP_HOST?

I am trying to find a way to test some code, but I need to rewrite a specific URL only from a specific HTTP_HOST The call goes out to http://SUB.DOMAIN.COM/showAssignment/7bde10b45efdd7a97629ef2fe01f7303/jsmodule/Nevow.Athena The ID in the middle is always random due to the cookie. I... (5 Replies)
Discussion started by: EXT3FSCK
5 Replies

5. Shell Programming and Scripting

[lynx dump] Order (by name/URL)

Hi :) How to use dump in lynx. $ lynx -dump http://www.google.com So, this is an example of a lynx dump: txt1 blabla Other txt some text 1. http://url_of_txt1 2. http://url_of_blabla 3. http://url_of_Other_txt 4. http://url_of_some_text ... How can i obtain this output? ... (12 Replies)
Discussion started by: aspire
12 Replies

6. UNIX for Dummies Questions & Answers

ReDirecting a URL to another URL - Linux

Hello, I need to redirect an existing URL, how can i do that? There's a current web address to a GUI that I have to redirect to another webaddress. Does anyone know how to do this? This is on Unix boxes Linux. example: https://m45.testing.address.net/host.php make it so the... (3 Replies)
Discussion started by: SkySmart
3 Replies

7. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

8. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

9. UNIX for Advanced & Expert Users

Posting a file from Unix to URl

HI Can you please help me,how to post a xml file from Unix to URL. Basically,i want to map contents of my file at an url Regards Pooja (1 Reply)
Discussion started by: PoojaM
1 Replies

10. UNIX for Dummies Questions & Answers

Replacing URL in a file with space

Hi, I have a file with a URL text written in it within double quotes e.g. "http://abcd.xyz.com/mno/somefile.dtd" I want the above text to get replaced by a single space character. I tried cat File1.txt | sed -e 's/("http)*(dtd")/ /g' > File2.txt But it didnt work out. Can someone... (5 Replies)
Discussion started by: dsrookie
5 Replies

Featured Tech Videos