Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Reading URL using Mechanize and dump all the contents of the URL to a file
Top Forums Shell Programming and Scripting Reading URL using Mechanize and dump all the contents of the URL to a file Post 302888744 by scott_cog on Monday 17th of February 2014 06:32:48 AM
Old 02-17-2014
Hi Ctsgnb,

Thanks for your reply.

Actually in the server we dont have wget ,curl utility and we cannot install as well.

Btw, i found the answer and i was able to get the output which i supposed to get. Hope it will be useful for others who are in search of such thing. and it was pretty simple.

Code:
#!/usr/bin/perl

use strict;
use warnings;
use LWP::UserAgent;

my $req;
my $res;

my $ua = LWP::UserAgent->new;
$ua->agent("Firefox/1.5.0.10");
$ua->timeout(1000);

$req = HTTP::Request->new(GET => 'http://yyyzzzzzxxxxxxxiiiii');

$res = $ua->request($req);

if ($res->is_success) {
    print $res->content;
}
else {
    print "Error: " . $res->status . "\n";
}

Last edited by scott_cog; 02-17-2014 at 11:22 AM.. Reason: .
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Replacing URL in a file with space

Hi, I have a file with a URL text written in it within double quotes e.g. "http://abcd.xyz.com/mno/somefile.dtd" I want the above text to get replaced by a single space character. I tried cat File1.txt | sed -e 's/("http)*(dtd")/ /g' > File2.txt But it didnt work out. Can someone... (5 Replies)
Discussion started by: dsrookie
5 Replies

2. UNIX for Advanced & Expert Users

Posting a file from Unix to URl

HI Can you please help me,how to post a xml file from Unix to URL. Basically,i want to map contents of my file at an url Regards Pooja (1 Reply)
Discussion started by: PoojaM
1 Replies

3. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

4. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

5. UNIX for Dummies Questions & Answers

ReDirecting a URL to another URL - Linux

Hello, I need to redirect an existing URL, how can i do that? There's a current web address to a GUI that I have to redirect to another webaddress. Does anyone know how to do this? This is on Unix boxes Linux. example: https://m45.testing.address.net/host.php make it so the... (3 Replies)
Discussion started by: SkySmart
3 Replies

6. Shell Programming and Scripting

[lynx dump] Order (by name/URL)

Hi :) How to use dump in lynx. $ lynx -dump http://www.google.com So, this is an example of a lynx dump: txt1 blabla Other txt some text 1. http://url_of_txt1 2. http://url_of_blabla 3. http://url_of_Other_txt 4. http://url_of_some_text ... How can i obtain this output? ... (12 Replies)
Discussion started by: aspire
12 Replies

7. Web Development

Regex to rewrite URL to another URL based on HTTP_HOST?

I am trying to find a way to test some code, but I need to rewrite a specific URL only from a specific HTTP_HOST The call goes out to http://SUB.DOMAIN.COM/showAssignment/7bde10b45efdd7a97629ef2fe01f7303/jsmodule/Nevow.Athena The ID in the middle is always random due to the cookie. I... (5 Replies)
Discussion started by: EXT3FSCK
5 Replies

8. Shell Programming and Scripting

Given a url how to get last part of file

HI, I have a URL that points to a file: LINK= "http://www.webpage.org/project/team2/file.tar" However when I try to use wget on this variable I receive the following error. wget $LINK line 4: http://www.webpage.org/project/team2/file.tar: No such file or directory wget:... (1 Reply)
Discussion started by: bashnewbee
1 Replies

9. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

10. Shell Programming and Scripting

Extracting the file name from the specified URL

Hello Everyone, I am trying to write a shell script(or Perl Script) that would do the following: I have a file that contains the following lines: File: https://ims-svnus.com/dev/DB/trunk/feeds/templates/shell_script.txt -r860... (5 Replies)
Discussion started by: filter
5 Replies

LEARN ABOUT MOJAVE

lwp::protocol::https

LWP::Protocol::https(3) 				User Contributed Perl Documentation				   LWP::Protocol::https(3)

NAME

       LWP::Protocol::https - Provide https support for LWP::UserAgent

SYNOPSIS

	 use LWP::UserAgent;

	 $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 });
	 $res = $ua->get("https://www.example.com");

DESCRIPTION

       The LWP::Protocol::https module provides support for using https schemed URLs with LWP.	This module is a plug-in to the LWP protocol
       handling, so you don't use it directly.	Once the module is installed LWP is able to access sites using HTTP over SSL/TLS.

       If hostname verification is requested by LWP::UserAgent's "ssl_opts", and neither "SSL_ca_file" nor "SSL_ca_path" is set, then
       "SSL_ca_file" is implied to be the one provided by Mozilla::CA.	If the Mozilla::CA module isn't available SSL requests will fail.  Either
       install this module, set up an alternative "SSL_ca_file" or disable hostname verification.

       This module used to be bundled with the libwww-perl, but it was unbundled in v6.02 in order to be able to declare its dependencies properly
       for the CPAN tool-chain.  Applications that need https support can just declare their dependency on LWP::Protocol::https and will no longer
       need to know what underlying modules to install.

SEE ALSO

       IO::Socket::SSL, Crypt::SSLeay, Mozilla::CA

COPYRIGHT

       Copyright 1997-2011 Gisle Aas.

       This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.18.2							    2013-04-29						   LWP::Protocol::https(3)
All times are GMT -4. The time now is 04:07 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy