Sponsored Content
Full Discussion: Wget -i URLs.txt problem
Top Forums UNIX for Dummies Questions & Answers Wget -i URLs.txt problem Post 302732633 by Corona688 on Sunday 18th of November 2012 09:28:19 PM
Old 11-18-2012
You can't just dump raw post data into a page that's not expecting it, it won't work. You need to look at the page and see what it's actually doing.

In this case, the login page is ( I think ) http://resell-rights-weekly.com/members/members.php, which expects post-data like you've been given.

To save the cookies, you need to tell wget to save the cookies(including session cookies), see man wget. So you could call wget once to login and save the cookies, and then call it again, loading the cookies to download the rest.
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sorting problem "sort -k 16,29 sample.txt > output.txt"

Hi all, Iam trying to sort the contents of the file based on the position of the file. Example: $cat sample.txt 0101020060731 ## Header record 1c1 Berger Awc ANP20070201301 4000.50 1c2 Bose W G ANP20070201609 6000.70 1c2 Andy CK ANP20070201230 28000.00... (3 Replies)
Discussion started by: ganapati
3 Replies

2. UNIX for Advanced & Expert Users

Wget FTP problem!

Hi, I've tried to download from ftp sites by wget but it failed and says "Service unavailable" but when I use sftp in binary mode and use "get" command it works perfectly. What's the problem? BTW: I tried both passive and active mode in wget. thnx for ur help (9 Replies)
Discussion started by: mjdousti
9 Replies

3. Shell Programming and Scripting

Problem with wget

Hi, I want to download some patches from SUN by using a script and I am using "wget" as the utillity for this. The website for downloading has a "https:" in its name as below https://sunsolve.sun.com/private-cgi/pdownload.pl?target=${line}&method=h and on running wget as below wget... (1 Reply)
Discussion started by: max29583
1 Replies

4. Shell Programming and Scripting

Extract urls from index.html downloaded using wget

Hi, I need to basically get a list of all the tarballs located at uri I am currently doing a wget on urito get the index.html page Now this index page contains the list of uris that I want to use in my bash script. can someone please guide me ,. I am new to Linux and shell scripting. ... (5 Replies)
Discussion started by: mnanavati
5 Replies

5. UNIX for Dummies Questions & Answers

Problem with wget no check certificate.

Hi, I'm trying to install some libraries, when running the makefile I get an error from the "wget --no check certificate option". I had a look help and the option wasn't listed. Anyone know what I'm missing. (0 Replies)
Discussion started by: davcra
0 Replies

6. UNIX for Dummies Questions & Answers

find lines in file1.txt not found in file2.txt memory problem

I have a diff command that does what I want but when comparing large text/log files, it uses up all the memory I have (sometimes over 8gig of memory) diff file1.txt file2.txt | grep '^<'| awk '{$1="";print $0}' | sed 's/^ *//' Is there a better more efficient way to find the lines in one file... (5 Replies)
Discussion started by: raptor25
5 Replies

7. Shell Programming and Scripting

Problem with wget and cookie

Dear people, I got a problem with an scrip using wget to download pdf-files from an website which uses session-cookies. Background: for university its quite nasty to look up weekly which new homeworks, papers etc. are available on the different sites of the universites chairs. So I wanted a... (1 Reply)
Discussion started by: jackomo
1 Replies

8. Shell Programming and Scripting

Download pdf's using wget convert to txt

wget -i genedx.txt The code above will download multiple pdf files from a site, but how can i download and convert these to .txt? I have attached the master list (genedx.txt - which contains the url and file names) as well as the two PDF's that are downloaded. I am trying to have those... (7 Replies)
Discussion started by: cmccabe
7 Replies

9. Proxy Server

Problem with wget

I cannot download anything using wget in centos 6.5 and 7. But I can update yum etc. # wget https://wordpress.org/latest.tar.gz --2014-10-23 13:50:23-- https://wordpress.org/latest.tar.gz Resolving wordpress.org... 66.155.40.249, 66.155.40.250 Connecting to wordpress.org|66.155.40.249|:443...... (3 Replies)
Discussion started by: nirosha
3 Replies
HTTP::Cookies(3)					User Contributed Perl Documentation					  HTTP::Cookies(3)

NAME
HTTP::Cookies - HTTP cookie jars SYNOPSIS
use HTTP::Cookies; $cookie_jar = HTTP::Cookies->new( file => "$ENV{'HOME'}/lwp_cookies.dat", autosave => 1, ); use LWP; my $browser = LWP::UserAgent->new; $browser->cookie_jar($cookie_jar); Or for an empty and temporary cookie jar: use LWP; my $browser = LWP::UserAgent->new; $browser->cookie_jar( {} ); DESCRIPTION
This class is for objects that represent a "cookie jar" -- that is, a database of all the HTTP cookies that a given LWP::UserAgent object knows about. Cookies are a general mechanism which server side connections can use to both store and retrieve information on the client side of the connection. For more information about cookies refer to <URL:http://curl.haxx.se/rfc/cookie_spec.html> and <URL:http://www.cookiecentral.com/>. This module also implements the new style cookies described in RFC 2965. The two variants of cookies are supposed to be able to coexist happily. Instances of the class HTTP::Cookies are able to store a collection of Set-Cookie2: and Set-Cookie: headers and are able to use this information to initialize Cookie-headers in HTTP::Request objects. The state of a HTTP::Cookies object can be saved in and restored from files. METHODS
The following methods are provided: $cookie_jar = HTTP::Cookies->new The constructor takes hash style parameters. The following parameters are recognized: file: name of the file to restore cookies from and save cookies to autosave: save during destruction (bool) ignore_discard: save even cookies that are requested to be discarded (bool) hide_cookie2: do not add Cookie2 header to requests Future parameters might include (not yet implemented): max_cookies 300 max_cookies_per_domain 20 max_cookie_size 4096 no_cookies list of domain names that we never return cookies to $cookie_jar->add_cookie_header( $request ) The add_cookie_header() method will set the appropriate Cookie:-header for the HTTP::Request object given as argument. The $request must have a valid url attribute before this method is called. $cookie_jar->extract_cookies( $response ) The extract_cookies() method will look for Set-Cookie: and Set-Cookie2: headers in the HTTP::Response object passed as argument. Any of these headers that are found are used to update the state of the $cookie_jar. $cookie_jar->set_cookie( $version, $key, $val, $path, $domain, $port, $path_spec, $secure, $maxage, $discard, \%rest ) The set_cookie() method updates the state of the $cookie_jar. The $key, $val, $domain, $port and $path arguments are strings. The $path_spec, $secure, $discard arguments are boolean values. The $maxage value is a number indicating number of seconds that this cookie will live. A value <= 0 will delete this cookie. %rest defines various other attributes like "Comment" and "CommentURL". $cookie_jar->save $cookie_jar->save( $file ) This method file saves the state of the $cookie_jar to a file. The state can then be restored later using the load() method. If a filename is not specified we will use the name specified during construction. If the attribute ignore_discard is set, then we will even save cookies that are marked to be discarded. The default is to save a sequence of "Set-Cookie3" lines. "Set-Cookie3" is a proprietary LWP format, not known to be compatible with any browser. The HTTP::Cookies::Netscape sub-class can be used to save in a format compatible with Netscape. $cookie_jar->load $cookie_jar->load( $file ) This method reads the cookies from the file and adds them to the $cookie_jar. The file must be in the format written by the save() method. $cookie_jar->revert This method empties the $cookie_jar and re-loads the $cookie_jar from the last save file. $cookie_jar->clear $cookie_jar->clear( $domain ) $cookie_jar->clear( $domain, $path ) $cookie_jar->clear( $domain, $path, $key ) Invoking this method without arguments will empty the whole $cookie_jar. If given a single argument only cookies belonging to that domain will be removed. If given two arguments, cookies belonging to the specified path within that domain are removed. If given three arguments, then the cookie with the specified key, path and domain is removed. $cookie_jar->clear_temporary_cookies Discard all temporary cookies. Scans for all cookies in the jar with either no expire field or a true "discard" flag. To be called when the user agent shuts down according to RFC 2965. $cookie_jar->scan( &callback ) The argument is a subroutine that will be invoked for each cookie stored in the $cookie_jar. The subroutine will be invoked with the following arguments: 0 version 1 key 2 val 3 path 4 domain 5 port 6 path_spec 7 secure 8 expires 9 discard 10 hash $cookie_jar->as_string $cookie_jar->as_string( $skip_discardables ) The as_string() method will return the state of the $cookie_jar represented as a sequence of "Set-Cookie3" header lines separated by " ". If $skip_discardables is TRUE, it will not return lines for cookies with the Discard attribute. SEE ALSO
HTTP::Cookies::Netscape, HTTP::Cookies::Microsoft COPYRIGHT
Copyright 1997-2002 Gisle Aas This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.18.2 2012-02-15 HTTP::Cookies(3)
All times are GMT -4. The time now is 09:59 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy