Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Reading URL using Mechanize and dump all the contents of the URL to a file
Top Forums Shell Programming and Scripting Reading URL using Mechanize and dump all the contents of the URL to a file Post 302888019 by scott_cog on Wednesday 12th of February 2014 10:03:35 AM
Old 02-12-2014
Reading URL using Mechanize and dump all the contents of the URL to a file
Hello,
Am very new to perl , please help me here !!

I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file.

below is the script which i have written so far ,

Code:
#!/usr/bin/perl
use LWP::UserAgent;
use WWW::Mechanize;
use WWW::DecodedContent;
use LWP::Debug qw( + );
use HTTP::Headers ;
my $mech = WWW::Mechanize->new( autocheck => 1 );
$mech->status();
print "Am inside Mechanize\n" ;

my $content = $mech->decoded_content || $mech->content;
#$mech->response()->decoded_content();
$ua->default_header("Accept-Encoding" => "gzip, deflate");
#$mech = WWW::Mechanize->new ;
$ua->agent("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
#$ua->default_header("Accept-Encoding" => "gzip, deflate");
$mech->get( "http://zzzzzzzzzURLzzzzzzzz" );
my $test = $mech->forms();
die "Can't even get the home page: ", $mech->response->status_line
unless $mech->success;

$mech->status();
$mech->uri();
 if ($response->is_success) {
   print "\n i am in finally";
   }
   else {
          die $response->status_line;
 }

Please feel free to provide some suggestion on changing the original script ,this is my first draft of code.

where as am getting an error " Can't call method "default_header" on an undefined value at getjan23.pl line 14. "

please let me know what am missing.

Thanks in advance
Last edited by scott_cog; 02-12-2014 at 11:05 AM.. Reason: .
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Replacing URL in a file with space

Hi, I have a file with a URL text written in it within double quotes e.g. "http://abcd.xyz.com/mno/somefile.dtd" I want the above text to get replaced by a single space character. I tried cat File1.txt | sed -e 's/("http)*(dtd")/ /g' > File2.txt But it didnt work out. Can someone... (5 Replies)
Discussion started by: dsrookie
5 Replies

2. UNIX for Advanced & Expert Users

Posting a file from Unix to URl

HI Can you please help me,how to post a xml file from Unix to URL. Basically,i want to map contents of my file at an url Regards Pooja (1 Reply)
Discussion started by: PoojaM
1 Replies

3. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

4. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

5. UNIX for Dummies Questions & Answers

ReDirecting a URL to another URL - Linux

Hello, I need to redirect an existing URL, how can i do that? There's a current web address to a GUI that I have to redirect to another webaddress. Does anyone know how to do this? This is on Unix boxes Linux. example: https://m45.testing.address.net/host.php make it so the... (3 Replies)
Discussion started by: SkySmart
3 Replies

6. Shell Programming and Scripting

[lynx dump] Order (by name/URL)

Hi :) How to use dump in lynx. $ lynx -dump http://www.google.com So, this is an example of a lynx dump: txt1 blabla Other txt some text 1. http://url_of_txt1 2. http://url_of_blabla 3. http://url_of_Other_txt 4. http://url_of_some_text ... How can i obtain this output? ... (12 Replies)
Discussion started by: aspire
12 Replies

7. Web Development

Regex to rewrite URL to another URL based on HTTP_HOST?

I am trying to find a way to test some code, but I need to rewrite a specific URL only from a specific HTTP_HOST The call goes out to http://SUB.DOMAIN.COM/showAssignment/7bde10b45efdd7a97629ef2fe01f7303/jsmodule/Nevow.Athena The ID in the middle is always random due to the cookie. I... (5 Replies)
Discussion started by: EXT3FSCK
5 Replies

8. Shell Programming and Scripting

Given a url how to get last part of file

HI, I have a URL that points to a file: LINK= "http://www.webpage.org/project/team2/file.tar" However when I try to use wget on this variable I receive the following error. wget $LINK line 4: http://www.webpage.org/project/team2/file.tar: No such file or directory wget:... (1 Reply)
Discussion started by: bashnewbee
1 Replies

9. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

10. Shell Programming and Scripting

Extracting the file name from the specified URL

Hello Everyone, I am trying to write a shell script(or Perl Script) that would do the following: I have a file that contains the following lines: File: https://ims-svnus.com/dev/DB/trunk/feeds/templates/shell_script.txt -r860... (5 Replies)
Discussion started by: filter
5 Replies

LEARN ABOUT DEBIAN

www::mechanize::formfiller::value::callback

WWW::Mechanize::FormFiller::Value::Callback(3pm)	User Contributed Perl Documentation	  WWW::Mechanize::FormFiller::Value::Callback(3pm)

NAME

       WWW::Mechanize::FormFiller::Value::Callback - Call Perl code to fill out a HTML form field

SYNOPSIS

	 use WWW::Mechanize::FormFiller;
	 use WWW::Mechanize::FormFiller::Value::Callback;

	 my $f = WWW::Mechanize::FormFiller->new();

	 # Create a default value for the HTML field "login"
	 # This will put the current login name into the login field

	 sub find_login {
	   getlogin || getpwuid($<) || "Kilroy";
	 };

	 my $login = WWW::Mechanize::FormFiller::Value::Callback->new( login => &find_login );
	 $f->add_value( login => $login );

	 # Alternatively take the following shorthand, which adds the
	 # field to the list as well :

	 # "If there is no password, put a nice number there
	 my $password = $f->add_filler( password => Callback => sub { int rand(90) + 10 } );

DESCRIPTION

       This class provides a way to write a value returned by Perl code into a HTML field.

       new NAME, CODE
	   Creates a new value which will correspond to the HTML field "NAME". The "CODE" is a code reference that will return the value to be
	   written into the HTML field.  The code reference will be called with two parameters, the object and the HTML::Form::Input object.

       name [NEWNAME]
	   Gets and sets the name of the HTML field this value corresponds to.

       value FIELD
	   Returns the value to put into the HTML field.

   EXPORT
       None by default.

   COPYRIGHT AND LICENSE
       This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

       Copyright (C) 2002,2003 Max Maischein

AUTHOR

       Max Maischein, <corion@cpan.org>

       Please contact me if you find bugs or otherwise improve the module. More tests are also very welcome !

SEE ALSO

       WWW::Mechanize, WWW::Mechanize::Shell, WWW::Mechanize::FormFiller, WWW::Mechanize::FormFiller::Value::Value,
       WWW::Mechanize::FormFiller::Value::Default, WWW::Mechanize::FormFiller::Value::Random, WWW::Mechanize::FormFiller::Value::Interactive

perl v5.10.1							    2009-04-24			  WWW::Mechanize::FormFiller::Value::Callback(3pm)
All times are GMT -4. The time now is 04:28 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy