Dear people, I got a problem with an scrip using wget to download pdf-files from an website which uses session-cookies.
Background: for university its quite nasty to look up weekly which new homeworks, papers etc. are available on the different sites of the universites chairs. So I wanted a script which download the homepage with all the links (wget), use sed to get the links and names of the dokuments out of the html page, and download the different documents with wget to the seperate folders.
Problem now: one website uses session cookies, you have to re-enter the Password every time (I think the cookie expires every day). thats my code to get the cookie:
I dont want to post the Page and password here public, if neccesary I can post the page, but maybe we can fix the problem without.
So I get out of this the cookie
I bet its a md5 sum including a timestamp.
Then I try to use the cookie.txt to download a file:
Problem, I only get redirekted to another page like when you entered a wrong password in the Browser (Firefox) and so would have got a wrong cookie/md5-sum.
Password is correct. When I readout the cookie out of firefox and copy and paste the md5-sum which is stored by firefox, it works perfectly! so my request for the cookie must be wrong.
On the website, the checking of username and Password is a normal html-form. I used firebug to edit the submit-part with "mailto:...." to be sure to get the correct strig which will be submitted by the browser. But entering this string into my shell-script to get wget to download the cookie didnt work.....(it generates two different md5-sum, thats why I estimate a time stamp is included , and otherwise a session-cookie wouldnt make sense. )
Ok, now has anybody an idea what do I do wrong?
Is there anything else helpful I can post you?
Its not an super-urgent matter, but it bothers me
thank you in advance!
jackomo
I cannot download anything using wget in centos 6.5 and 7. But I can update yum etc.
# wget https://wordpress.org/latest.tar.gz
--2014-10-23 13:50:23-- https://wordpress.org/latest.tar.gz
Resolving wordpress.org... 66.155.40.249, 66.155.40.250
Connecting to wordpress.org|66.155.40.249|:443...... (3 Replies)
I am trying to wget to retrieve a file from an ubuntu server. I believe I need a cookie authentication, but am am having trouble.
I am reading the maual and it is not making sense. I have also tried
wget http://..... --username=... --password=... Thank you. (2 Replies)
Hi Everyone,
I have a problem with wget using an input file of URLs. When I execute this -> wget -i URLs.txt I get the login.php pages transferred but not the files I have in the URLs.txt file. I need to use the input file because it will have new products to download each week. I want my VA to... (3 Replies)
Hi, I'm trying to install some libraries, when running the makefile I get an error from the "wget --no check certificate option". I had a look help and the option wasn't listed. Anyone know what I'm missing. (0 Replies)
How can I create a cookie with expiration date using either wget or curl?
I am able to use cli wget, cli curl, or php curl.
I don't see that the cookies I have created contain an expiration date, and I have a problem logging into a forum that looks for the expiration date. (2 Replies)
Hi. I am trying to make a mirror of this free online journal:
http://www.informaworld.com/smpp/title~content=t716100758~db=all
Under the individual issues, the link location for the "Full Text PDF" does not have ".pdf" as an extension -- so when I use wget it misses the file. However clicking... (5 Replies)
Hi Everyone,
I am facing the problem to create the cookie in CGI (bash script).
Is it possible can create in cgi? or javascript better?
Anyone got the sample to create the cookie in cgi(bash script)?
Just the login will do ->> USERNAME and PASSWORD
after create how to store into the... (2 Replies)
Hi,
I want to download some patches from SUN by using a script and I am using "wget" as the utillity for this.
The website for downloading has a "https:" in its name as below
https://sunsolve.sun.com/private-cgi/pdownload.pl?target=${line}&method=h
and on running wget as below
wget... (1 Reply)
Hi,
I've tried to download from ftp sites by wget but it failed and says "Service unavailable" but when I use sftp in binary mode and use "get" command it works perfectly. What's the problem?
BTW: I tried both passive and active mode in wget.
thnx for ur help (9 Replies)
hi frnds
i m getting this error whenever i m trying to run otrs in my web browser.
http://ipaddr /otrs/index.pl
"
500 Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your... (1 Reply)