Sponsored Content
Top Forums Shell Programming and Scripting Reading URL using Mechanize and dump all the contents of the URL to a file Post 302888400 by ctsgnb on Friday 14th of February 2014 05:43:59 AM
Old 02-14-2014
Check the wget utility, i guess it does what you need ... without having to reinvent the wheel Smilie
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Replacing URL in a file with space

Hi, I have a file with a URL text written in it within double quotes e.g. "http://abcd.xyz.com/mno/somefile.dtd" I want the above text to get replaced by a single space character. I tried cat File1.txt | sed -e 's/("http)*(dtd")/ /g' > File2.txt But it didnt work out. Can someone... (5 Replies)
Discussion started by: dsrookie
5 Replies

2. UNIX for Advanced & Expert Users

Posting a file from Unix to URl

HI Can you please help me,how to post a xml file from Unix to URL. Basically,i want to map contents of my file at an url Regards Pooja (1 Reply)
Discussion started by: PoojaM
1 Replies

3. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

4. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

5. UNIX for Dummies Questions & Answers

ReDirecting a URL to another URL - Linux

Hello, I need to redirect an existing URL, how can i do that? There's a current web address to a GUI that I have to redirect to another webaddress. Does anyone know how to do this? This is on Unix boxes Linux. example: https://m45.testing.address.net/host.php make it so the... (3 Replies)
Discussion started by: SkySmart
3 Replies

6. Shell Programming and Scripting

[lynx dump] Order (by name/URL)

Hi :) How to use dump in lynx. $ lynx -dump http://www.google.com So, this is an example of a lynx dump: txt1 blabla Other txt some text 1. http://url_of_txt1 2. http://url_of_blabla 3. http://url_of_Other_txt 4. http://url_of_some_text ... How can i obtain this output? ... (12 Replies)
Discussion started by: aspire
12 Replies

7. Web Development

Regex to rewrite URL to another URL based on HTTP_HOST?

I am trying to find a way to test some code, but I need to rewrite a specific URL only from a specific HTTP_HOST The call goes out to http://SUB.DOMAIN.COM/showAssignment/7bde10b45efdd7a97629ef2fe01f7303/jsmodule/Nevow.Athena The ID in the middle is always random due to the cookie. I... (5 Replies)
Discussion started by: EXT3FSCK
5 Replies

8. Shell Programming and Scripting

Given a url how to get last part of file

HI, I have a URL that points to a file: LINK= "http://www.webpage.org/project/team2/file.tar" However when I try to use wget on this variable I receive the following error. wget $LINK line 4: http://www.webpage.org/project/team2/file.tar: No such file or directory wget:... (1 Reply)
Discussion started by: bashnewbee
1 Replies

9. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

10. Shell Programming and Scripting

Extracting the file name from the specified URL

Hello Everyone, I am trying to write a shell script(or Perl Script) that would do the following: I have a file that contains the following lines: File: https://ims-svnus.com/dev/DB/trunk/feeds/templates/shell_script.txt -r860... (5 Replies)
Discussion started by: filter
5 Replies
HTTP::Recorder(3pm)					User Contributed Perl Documentation				       HTTP::Recorder(3pm)

NAME
HTTP::Recorder - record interaction with websites SYNOPSIS
Using HTTP::Recorder as a Web Proxy Set HTTP::Recorder as the user agent for a proxy, and it rewrites HTTP responses so that additional requests can be recorded. The Proxy Script For quick start, run the httprecorder script httprecorder This will open a local proxy on port 8080, and will dump the recorded traffic to a file named http_traffic in the current directory. use the -help parameter for usage info Start the proxy script, then change the settings in your web browser so that it will use this proxy for web requests. For more information about proxy settings and the default port, see HTTP::Proxy. The script will be recorded in the specified file, and can be viewed and modified via the control panel. For better control, use this example: #!/usr/bin/perl use HTTP::Proxy; use HTTP::Recorder; my $proxy = HTTP::Proxy->new(); # create a new HTTP::Recorder object my $agent = new HTTP::Recorder; # set the log file (optional) $agent->file("/tmp/myfile"); # set HTTP::Recorder as the agent for the proxy $proxy->agent( $agent ); # start the proxy $proxy->start(); Start Recording Now you can use your browser as your normally would, and your actions will be recorded in the file you specified. Alternatively, you can start recording from the Control Panel. Using the Control Panel If you have Javascript enabled in your browser, go to the HTTP::Recorder control URL (http://http-recorder by default), optionally type a URL into the "Goto page" field, and click "Go". In the new window, interact with web sites as you normally do, including typing a new address into the address field. The Control Panel will be updated after each recorded action. The Control Panel allows you to modify, delete, or save your script. SSL sessions As of version 0.03, HTTP::Recorder can record SSL sessions. To begin recording an SSL session, go to the control URL (http://http-recorder/ by default), and enter the initial URL. Then, interact with the web site as usual. Script output By default, HTTP::Recorder outputs WWW::Mechanize scripts. However, you can override HTTP::Recorder::Logger to output other types of scripts. Functions new Creates and returns a new HTTP::Recorder object, referred to as the 'agent'. $agent->prefix([$value]) Get or set the prefix string that HTTP::Recorder uses for rewriting responses. $agent->control([$value]) Get or set the URL of the control panel. By default, the control URL is 'http-recorder'. The control URL will display a control panel which will allow you to view and edit the current script. $agent->logger([$value]) Get or set the logger object. The default logger is a HTTP::Recorder::Logger, which generates WWW::Mechanize scripts. $agent->ignore_favicon([0|1]) Get or set ignore_favicon flag that causes HTTP::Recorder to skip logging requests favicon.ico files. The value is 1 by default. $agent->file([$value]) Get or set the filename for generated scripts. The default is '/tmp/scriptfile'. Bugs, Missing Features, and other Oddities Javascript WWW::Mechanize can't play back Javascript actions, and HTTP::Recorder doesn't record them. Why are my images corrupted? HTTP::Recorder only tries to rewrite responses that are of type text/*, which it determines by reading the Content-Type header of the HTTP::Response object. However, if the received image gives the wrong Content-Type header, it may be corrupted by the recorder. While this may not be pleasant to look at, it shouldn't have an effect on your recording session. See Also See also LWP::UserAgent, WWW::Mechanize, HTTP::Proxy. Requests &; Bugs Please submit any feature requests, suggestions, bugs, or patches at http://rt.cpan.org/, or email to bug-HTTP-Recorder@rt.cpan.org. If you're submitting a bug of the type "X doesn't record correctly," be sure to include a (preferably short and simple) HTML page that demonstrates the problem, and a clear explanation of a) what it does that it shouldn't, and b) what it should do instead. Author Copyright 2003-2005 by Linda Julien <leira@cpan.org> Maintained by Shmuel Fomberg <semuelf@cpan.org> Released under the GNU Public License. perl v5.14.2 2012-04-23 HTTP::Recorder(3pm)
All times are GMT -4. The time now is 11:04 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy