Sponsored Content
Top Forums Shell Programming and Scripting help with wget and 404 errors Post 302687515 by problemss on Thursday 16th of August 2012 05:19:48 PM
Old 08-16-2012
@fpmurphy do you know of anything to convert a dynamic page to static and store it locally? The reason i want to store this locally is because 1) the site causes timeouts with some scripts i have 2) I do no want to overload or be the cause of extra traffic on the server because of my scripts

@Corona688 I added the user agent, but already had a referer. It still fails. My query looks like this:

Code:
wget --directory-prefix=/Users/problemss/Desktop --proxy=off -Q0 --user-agent=Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) --passive-ftp --header=REFERER:http://accuscore.com -k -r -l2 --progress=dot:binary http://accuscore.com/fantasy-sports/nfl-fantasy-sports/Current-Week-DEF-ST

The response is:

Code:
Resolving accuscore.com... 184.106.172.20
Connecting to accuscore.com|184.106.172.20|:80... connected.
HTTP request sent, awaiting response... 404 NOT FOUND
2012-08-16 14:13:25 ERROR 404: NOT FOUND.

 

8 More Discussions You Might Find Interesting

1. Web Development

mod_rewrite RewriteMap - possible to 404?

In my strenuous efforts to get SEO-friendly urls, I'm using a rewrite map in my apache setup: RewriteEngine on RewriteOptions MaxRedirects=5 RewriteMap seo prg:/Applications/MAMP/htdocs/map.php #map requests for the original file to the new SEO friendly urls RewriteCond... (0 Replies)
Discussion started by: sneakyimp
0 Replies

2. Web Development

HTTP 404 Error Fetches File from Another Server

Has any seen any PHP or other scripting code that will executive on a 404 "File Not Found' error and then fetch the requested file from a different server? (0 Replies)
Discussion started by: Neo
0 Replies

3. Web Development

[.htaccess] Denegar IP Con Error 404

Saludos amigos programadores de Web. Mi Problema es, que necesito denegar una IP desde el fichero .htaccess, pero no con el tipico error 403 (Forbidden). sino con el error 404 (Not found). Para quitarme de encima una IP fija que visita constantemente mi servidor para llenar de Spam mis... (1 Reply)
Discussion started by: Ignacio A
1 Replies

4. Red Hat

HTTP/1.1 404 Not Found error in Web Server

I am running 2 jboss instances with ports 8585 and 8686 in my web server. Now trying to get header using the command curl -s --connect-timeout 360 -m 360 --head http: // localhost:8686/ then i get the following error HTTP/1.1 404 Not Found Server: Apache-Coyote/1.1 Content-Length: 0... (1 Reply)
Discussion started by: hridan
1 Replies

5. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403... (2 Replies)
Discussion started by: Prasannag87
2 Replies

6. What is on Your Mind?

New Responsive 404 Page for UNIX.com

Just created (actually, only modified... it was created by ShoutOut) a new responsive 404 "not found" page with the help of ShoutOut free templates. https://www.unix.com/status/404.html Same for 401 and 403 errors. Picture sans animation: ... (2 Replies)
Discussion started by: Neo
2 Replies

7. What is on Your Mind?

YouTube: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search

Getting a bit more comfortable making quick YT videos in 4K, here is: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search Console https://youtu.be/I6b9T2qcqFo (0 Replies)
Discussion started by: Neo
0 Replies

8. What is on Your Mind?

Google Webmaster Tools Shows Problems with Soft 404 Errors

Well, Sorry, but I cannot seem to fix the problem with the steady decline of ranking for unix.com pages with Google. Google Webmaster Tools show that they are dropping our pages from the indexes more and more because of "Soft 404" errors which started after we moved to the new data center. ... (18 Replies)
Discussion started by: Neo
18 Replies
HTTP::Recorder(3pm)					User Contributed Perl Documentation				       HTTP::Recorder(3pm)

NAME
HTTP::Recorder - record interaction with websites SYNOPSIS
Using HTTP::Recorder as a Web Proxy Set HTTP::Recorder as the user agent for a proxy, and it rewrites HTTP responses so that additional requests can be recorded. The Proxy Script For quick start, run the httprecorder script httprecorder This will open a local proxy on port 8080, and will dump the recorded traffic to a file named http_traffic in the current directory. use the -help parameter for usage info Start the proxy script, then change the settings in your web browser so that it will use this proxy for web requests. For more information about proxy settings and the default port, see HTTP::Proxy. The script will be recorded in the specified file, and can be viewed and modified via the control panel. For better control, use this example: #!/usr/bin/perl use HTTP::Proxy; use HTTP::Recorder; my $proxy = HTTP::Proxy->new(); # create a new HTTP::Recorder object my $agent = new HTTP::Recorder; # set the log file (optional) $agent->file("/tmp/myfile"); # set HTTP::Recorder as the agent for the proxy $proxy->agent( $agent ); # start the proxy $proxy->start(); Start Recording Now you can use your browser as your normally would, and your actions will be recorded in the file you specified. Alternatively, you can start recording from the Control Panel. Using the Control Panel If you have Javascript enabled in your browser, go to the HTTP::Recorder control URL (http://http-recorder by default), optionally type a URL into the "Goto page" field, and click "Go". In the new window, interact with web sites as you normally do, including typing a new address into the address field. The Control Panel will be updated after each recorded action. The Control Panel allows you to modify, delete, or save your script. SSL sessions As of version 0.03, HTTP::Recorder can record SSL sessions. To begin recording an SSL session, go to the control URL (http://http-recorder/ by default), and enter the initial URL. Then, interact with the web site as usual. Script output By default, HTTP::Recorder outputs WWW::Mechanize scripts. However, you can override HTTP::Recorder::Logger to output other types of scripts. Functions new Creates and returns a new HTTP::Recorder object, referred to as the 'agent'. $agent->prefix([$value]) Get or set the prefix string that HTTP::Recorder uses for rewriting responses. $agent->control([$value]) Get or set the URL of the control panel. By default, the control URL is 'http-recorder'. The control URL will display a control panel which will allow you to view and edit the current script. $agent->logger([$value]) Get or set the logger object. The default logger is a HTTP::Recorder::Logger, which generates WWW::Mechanize scripts. $agent->ignore_favicon([0|1]) Get or set ignore_favicon flag that causes HTTP::Recorder to skip logging requests favicon.ico files. The value is 1 by default. $agent->file([$value]) Get or set the filename for generated scripts. The default is '/tmp/scriptfile'. Bugs, Missing Features, and other Oddities Javascript WWW::Mechanize can't play back Javascript actions, and HTTP::Recorder doesn't record them. Why are my images corrupted? HTTP::Recorder only tries to rewrite responses that are of type text/*, which it determines by reading the Content-Type header of the HTTP::Response object. However, if the received image gives the wrong Content-Type header, it may be corrupted by the recorder. While this may not be pleasant to look at, it shouldn't have an effect on your recording session. See Also See also LWP::UserAgent, WWW::Mechanize, HTTP::Proxy. Requests &; Bugs Please submit any feature requests, suggestions, bugs, or patches at http://rt.cpan.org/, or email to bug-HTTP-Recorder@rt.cpan.org. If you're submitting a bug of the type "X doesn't record correctly," be sure to include a (preferably short and simple) HTML page that demonstrates the problem, and a clear explanation of a) what it does that it shouldn't, and b) what it should do instead. Author Copyright 2003-2005 by Linda Julien <leira@cpan.org> Maintained by Shmuel Fomberg <semuelf@cpan.org> Released under the GNU Public License. perl v5.14.2 2012-04-23 HTTP::Recorder(3pm)
All times are GMT -4. The time now is 11:57 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy