Sponsored Content
Top Forums Shell Programming and Scripting Reading URL using Mechanize and dump all the contents of the URL to a file Post 302888019 by scott_cog on Wednesday 12th of February 2014 10:03:35 AM
Old 02-12-2014
Reading URL using Mechanize and dump all the contents of the URL to a file

Hello,
Am very new to perl , please help me here !!

I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file.

below is the script which i have written so far ,

Code:
#!/usr/bin/perl
use LWP::UserAgent;
use WWW::Mechanize;
use WWW::DecodedContent;
use LWP::Debug qw( + );
use HTTP::Headers ;
my $mech = WWW::Mechanize->new( autocheck => 1 );
$mech->status();
print "Am inside Mechanize\n" ;

my $content = $mech->decoded_content || $mech->content;
#$mech->response()->decoded_content();
$ua->default_header("Accept-Encoding" => "gzip, deflate");
#$mech = WWW::Mechanize->new ;
$ua->agent("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
#$ua->default_header("Accept-Encoding" => "gzip, deflate");
$mech->get( "http://zzzzzzzzzURLzzzzzzzz" );
my $test = $mech->forms();
die "Can't even get the home page: ", $mech->response->status_line
unless $mech->success;

$mech->status();
$mech->uri();
 if ($response->is_success) {
   print "\n i am in finally";
   }
   else {
          die $response->status_line;
 }

Please feel free to provide some suggestion on changing the original script ,this is my first draft of code.

where as am getting an error " Can't call method "default_header" on an undefined value at getjan23.pl line 14. "

please let me know what am missing.

Thanks in advance

Last edited by scott_cog; 02-12-2014 at 11:05 AM.. Reason: .
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Replacing URL in a file with space

Hi, I have a file with a URL text written in it within double quotes e.g. "http://abcd.xyz.com/mno/somefile.dtd" I want the above text to get replaced by a single space character. I tried cat File1.txt | sed -e 's/("http)*(dtd")/ /g' > File2.txt But it didnt work out. Can someone... (5 Replies)
Discussion started by: dsrookie
5 Replies

2. UNIX for Advanced & Expert Users

Posting a file from Unix to URl

HI Can you please help me,how to post a xml file from Unix to URL. Basically,i want to map contents of my file at an url Regards Pooja (1 Reply)
Discussion started by: PoojaM
1 Replies

3. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

4. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

5. UNIX for Dummies Questions & Answers

ReDirecting a URL to another URL - Linux

Hello, I need to redirect an existing URL, how can i do that? There's a current web address to a GUI that I have to redirect to another webaddress. Does anyone know how to do this? This is on Unix boxes Linux. example: https://m45.testing.address.net/host.php make it so the... (3 Replies)
Discussion started by: SkySmart
3 Replies

6. Shell Programming and Scripting

[lynx dump] Order (by name/URL)

Hi :) How to use dump in lynx. $ lynx -dump http://www.google.com So, this is an example of a lynx dump: txt1 blabla Other txt some text 1. http://url_of_txt1 2. http://url_of_blabla 3. http://url_of_Other_txt 4. http://url_of_some_text ... How can i obtain this output? ... (12 Replies)
Discussion started by: aspire
12 Replies

7. Web Development

Regex to rewrite URL to another URL based on HTTP_HOST?

I am trying to find a way to test some code, but I need to rewrite a specific URL only from a specific HTTP_HOST The call goes out to http://SUB.DOMAIN.COM/showAssignment/7bde10b45efdd7a97629ef2fe01f7303/jsmodule/Nevow.Athena The ID in the middle is always random due to the cookie. I... (5 Replies)
Discussion started by: EXT3FSCK
5 Replies

8. Shell Programming and Scripting

Given a url how to get last part of file

HI, I have a URL that points to a file: LINK= "http://www.webpage.org/project/team2/file.tar" However when I try to use wget on this variable I receive the following error. wget $LINK line 4: http://www.webpage.org/project/team2/file.tar: No such file or directory wget:... (1 Reply)
Discussion started by: bashnewbee
1 Replies

9. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

10. Shell Programming and Scripting

Extracting the file name from the specified URL

Hello Everyone, I am trying to write a shell script(or Perl Script) that would do the following: I have a file that contains the following lines: File: https://ims-svnus.com/dev/DB/trunk/feeds/templates/shell_script.txt -r860... (5 Replies)
Discussion started by: filter
5 Replies
Test::WWW::Mechanize::CGIApp(3pm)			User Contributed Perl Documentation			 Test::WWW::Mechanize::CGIApp(3pm)

NAME
Test::WWW::Mechanize::CGIApp - Test::WWW::Mechanize for CGI::Application SYNOPSIS
# We're in a t/*.t test script... use Test::WWW::Mechanize::CGIApp; my $mech = Test::WWW::Mechanize::CGIApp->new; # test a class that uses CGI::Application calling semantics. # (in this case we'll new up an instance of the app and call # its ->run() method) # $mech->app("My::WebApp"); $mech->get_ok("?rm=my_run_mode&arg1=1&arg2=42"); # test a class that uses CGI::Application::Dispatch # to locate the run_mode # (in this case we'll just call the ->dispatch() class method). # my $dispatched_mech = Test::WWW::Mechanize::CGIApp->new; $dispatched_mech->app("My::DispatchApp"); $mech->get_ok("/WebApp/my_run_mode?arg1=1&arg2=42"); # create an anonymous sub that this class will use to # handle the request. # # this could be useful if you need to do something novel # after creating an instance of your class (e.g. the # fiddle_with_stuff() below) or maybe you have a unique # way to get the app to run. # my $custom_mech = Test::WWW::Mechanize::CGIApp->new; $custom_mech->app( sub { require "My::WebApp"; my $app = My::WebApp->new(); $app->fiddle_with_stuff(); $app->run(); }); $mech->get_ok("?rm=my_run_mode&arg1=1&arg2=42"); # at this point you can play with all kinds of cool # Test::WWW::Mechanize testing methods. is($mech->ct, "text/html"); $mech->title_is("Root", "On the root page"); $mech->content_contains("This is the root page", "Correct content"); $mech->follow_link_ok({text => 'Hello'}, "Click on Hello"); # ... and all other Test::WWW::Mechanize methods DESCRIPTION
This package makes testing CGIApp based modules fast and easy. It takes advantage of Test::WWW::Mechanize to provide functions for common web testing scenarios. For example: $mech->get_ok( $page ); $mech->title_is( "Invoice Status", "Make sure we're on the invoice page" ); $mech->content_contains( "Andy Lester", "My name somewhere" ); $mech->content_like( qr/(cpan|perl).org/, "Link to perl.org or CPAN" ); For applications that inherit from CGI::Application it will handle requests by creating a new instance of the class and calling its "run" method. For applications that use CGI::Application::Dispatch it will call the "dispatch" class method. If neither of these options are the right thing, you can set a reference to a sub that will be used to handle the request. This module supports cookies automatically. Check out Test::WWW::Mechanize for more information about all of the cool things you can test! CONSTRUCTOR
new Behaves like, and calls, Test::WWW::Mechanize's "new" method. It optionally uses an "app" parameter (see below), any other parameters get passed to Test::WWW::Mechanize's constructor. Note that you can either pass the name of the CGI::Application into the constructor using the "app" parameter or set it later using the "app" method. use Test::WWW::Mechanize::CGIApp; my $mech = Test::WWW::Mechanize::CGIApp->new; # or my $mech = Test::WWW::Mechanize::CGIApp->new(app => 'TestApp'); METHODS
$mech->app($app_handler) This method provides a mechanism for informing Test::WWW::Mechanize::CGIApp how it should go about executing your run_mode. If you set it to the name of a class, then it will load the class and either create an instance and ->run() it (if it's CGI::Application based), invoke the ->dispatch() method if it's CGI::Application::Dispatch based, or call the supplied anonymous subroutine and let it do all of the heavy lifting. SEE ALSO
Related modules which may be of interest: Test::WWW::Mechanize, WWW::Mechanize. Various implementation tricks came from Test::WWW::Mechanize::Catalyst. AUTHOR
George Hartzell, "<hartzell@alerce.com>" based on Test::WWW::Mechanize::Catalyst by Leon Brocard, "<acme@astray.com>". COPYRIGHT
Copyright (C) 2007, George Hartzell This module is free software; you can redistribute it or modify it under the same terms as Perl itself. perl v5.8.8 2008-03-12 Test::WWW::Mechanize::CGIApp(3pm)
All times are GMT -4. The time now is 05:09 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy