Sponsored Content
Full Discussion: Problem with wget and cookie
Top Forums Shell Programming and Scripting Problem with wget and cookie Post 302661027 by jackomo on Sunday 24th of June 2012 12:39:03 PM
Old 06-24-2012
Java Problem with wget and cookie

Dear people, I got a problem with an scrip using wget to download pdf-files from an website which uses session-cookies.

Background: for university its quite nasty to look up weekly which new homeworks, papers etc. are available on the different sites of the universites chairs. So I wanted a script which download the homepage with all the links (wget), use sed to get the links and names of the dokuments out of the html page, and download the different documents with wget to the seperate folders.

Problem now: one website uses session cookies, you have to re-enter the Password every time (I think the cookie expires every day). thats my code to get the cookie:
Code:
#!/bin/sh

# Log in to the server.  This can be done only once.
          wget --save-cookies cookies.txt \
               --post-data 'successRedirect=http%3A%2F%2Fwww.xxxxxxxxxxxxxx.de%2F%2Ffile.php%3Fid%3D20745%26l%3D1%26&usrName=XXXX&usrPass=XXXXX&login=Anmelden' \
        --keep-session-cookies \
               http://www.xxxxxxxxxxxxxx.de//index.php?id=20745&o=18059&_f=20745&l=1&_action=login

I dont want to post the Page and password here public, if neccesary I can post the page, but maybe we can fix the problem without.

So I get out of this the cookie
Code:
# HTTP cookie file.
# Generated by Wget on 2012-06-21 22:26:04.
# Edit at your own risk.

www.xxxxxxxxx.de    FALSE    /    FALSE    0    PHPSESSID    cc124cb622ffa9fd60f288a68db3e597

I bet its a md5 sum including a timestamp.

Then I try to use the cookie.txt to download a file:
Code:
# Now grab the page or pages we care about.
          wget --load-cookies cookies.txt \
               -p http://www.xxxxxxxx.de//file.php?id=20745

Problem, I only get redirekted to another page like when you entered a wrong password in the Browser (Firefox) and so would have got a wrong cookie/md5-sum.

Password is correct. When I readout the cookie out of firefox and copy and paste the md5-sum which is stored by firefox, it works perfectly! so my request for the cookie must be wrong.

On the website, the checking of username and Password is a normal html-form. I used firebug to edit the submit-part with "mailto:...." to be sure to get the correct strig which will be submitted by the browser. But entering this string into my shell-script to get wget to download the cookie didnt work.....(it generates two different md5-sum, thats why I estimate a time stamp is included Smilie, and otherwise a session-cookie wouldnt make sense. )

Ok, now has anybody an idea what do I do wrong?
Is there anything else helpful I can post you?
Its not an super-urgent matter, but it bothers me Smilie
thank you in advance!
jackomo
 

10 More Discussions You Might Find Interesting

1. Linux

Cant locate CGI/Cookie.pm

hi frnds i m getting this error whenever i m trying to run otrs in my web browser. http://ipaddr /otrs/index.pl " 500 Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your... (1 Reply)
Discussion started by: naik_mit
1 Replies

2. UNIX for Advanced & Expert Users

Wget FTP problem!

Hi, I've tried to download from ftp sites by wget but it failed and says "Service unavailable" but when I use sftp in binary mode and use "get" command it works perfectly. What's the problem? BTW: I tried both passive and active mode in wget. thnx for ur help (9 Replies)
Discussion started by: mjdousti
9 Replies

3. Shell Programming and Scripting

Problem with wget

Hi, I want to download some patches from SUN by using a script and I am using "wget" as the utillity for this. The website for downloading has a "https:" in its name as below https://sunsolve.sun.com/private-cgi/pdownload.pl?target=${line}&method=h and on running wget as below wget... (1 Reply)
Discussion started by: max29583
1 Replies

4. Shell Programming and Scripting

problem for CGI create Cookie!!!!

Hi Everyone, I am facing the problem to create the cookie in CGI (bash script). Is it possible can create in cgi? or javascript better? Anyone got the sample to create the cookie in cgi(bash script)? Just the login will do ->> USERNAME and PASSWORD after create how to store into the... (2 Replies)
Discussion started by: ryanW
2 Replies

5. UNIX for Dummies Questions & Answers

wget pdf downloading problem

Hi. I am trying to make a mirror of this free online journal: http://www.informaworld.com/smpp/title~content=t716100758~db=all Under the individual issues, the link location for the "Full Text PDF" does not have ".pdf" as an extension -- so when I use wget it misses the file. However clicking... (5 Replies)
Discussion started by: obo1234
5 Replies

6. Shell Programming and Scripting

wget - cookie with expiration date

How can I create a cookie with expiration date using either wget or curl? I am able to use cli wget, cli curl, or php curl. I don't see that the cookies I have created contain an expiration date, and I have a problem logging into a forum that looks for the expiration date. (2 Replies)
Discussion started by: locoroco
2 Replies

7. UNIX for Dummies Questions & Answers

Problem with wget no check certificate.

Hi, I'm trying to install some libraries, when running the makefile I get an error from the "wget --no check certificate option". I had a look help and the option wasn't listed. Anyone know what I'm missing. (0 Replies)
Discussion started by: davcra
0 Replies

8. UNIX for Dummies Questions & Answers

Wget -i URLs.txt problem

Hi Everyone, I have a problem with wget using an input file of URLs. When I execute this -> wget -i URLs.txt I get the login.php pages transferred but not the files I have in the URLs.txt file. I need to use the input file because it will have new products to download each week. I want my VA to... (3 Replies)
Discussion started by: Keith londrie
3 Replies

9. Shell Programming and Scripting

Cookie authenticationn

I am trying to wget to retrieve a file from an ubuntu server. I believe I need a cookie authentication, but am am having trouble. I am reading the maual and it is not making sense. I have also tried wget http://..... --username=... --password=... Thank you. (2 Replies)
Discussion started by: cmccabe
2 Replies

10. Proxy Server

Problem with wget

I cannot download anything using wget in centos 6.5 and 7. But I can update yum etc. # wget https://wordpress.org/latest.tar.gz --2014-10-23 13:50:23-- https://wordpress.org/latest.tar.gz Resolving wordpress.org... 66.155.40.249, 66.155.40.250 Connecting to wordpress.org|66.155.40.249|:443...... (3 Replies)
Discussion started by: nirosha
3 Replies
SMBGET(1)							   User Commands							 SMBGET(1)

NAME
smbget - wget-like utility for download files over SMB SYNOPSIS
smbget [-a, --guest] [-r, --resume] [-R, --recursive] [-u, --username=STRING] [-p, --password=STRING] [-w, --workgroup=STRING] [-n, --nonprompt] [-d, --debuglevel=INT] [-D, --dots] [-P, --keep-permissions] [-o, --outputfile] [-f, --rcfile] [-q, --quiet] [-v, --verbose] [-b, --blocksize] [-O, --stdout] [-?, --help] [--usage] {smb://host/share/path/to/file} [smb://url2/] [...] DESCRIPTION
This tool is part of the samba(7) suite. smbget is a simple utility with wget-like semantics, that can download files from SMB servers. You can specify the files you would like to download on the command-line. The files should be in the smb-URL standard, e.g. use smb://host/share/file for the UNC path \\HOST\SHARE\file. OPTIONS
-a, --guest Work as user guest -r, --resume Automatically resume aborted files -R, --recursive Recursively download files -u, --username=STRING Username to use -p, --password=STRING Password to use -w, --workgroup=STRING Workgroup to use (optional) -n, --nonprompt Don't ask anything (non-interactive) -d, --debuglevel=INT Debuglevel to use -D, --dots Show dots as progress indication -P, --keep-permissions Set same permissions on local file as are set on remote file. -o, --outputfile Write the file that is being downloaded to the specified file. Can not be used together with -R. -O, --stdout Write the file that is being downloaded to standard output. -f, --rcfile Use specified rcfile. This will be loaded in the order it was specified - e.g. if you specify any options before this one, they might get overriden by the contents of the rcfile. -q, --quiet Be quiet -v, --verbose Be verbose -b, --blocksize Number of bytes to download in a block. Defaults to 64000. -?, --help Show help message --usage Display brief usage message SMB URLS
SMB URL's should be specified in the following format: smb://[[[domain;]user[:password@]]server[/share[/path[/file]]]] smb:// means all the workgroups smb://name/ means, if name is a workgroup, all the servers in this workgroup, or if name is a server, all the shares on this server. EXAMPLES
# Recursively download 'src' directory smbget -R smb://rhonwyn/jelmer/src # Download FreeBSD ISO and enable resuming smbget -r smb://rhonwyn/isos/FreeBSD5.1.iso # Recursively download all ISOs smbget -Rr smb://rhonwyn/isos # Backup my data on rhonwyn smbget -Rr smb://rhonwyn/ BUGS
Permission denied is returned in some cases where the cause of the error is unknown (such as an illegally formatted smb:// url or trying to get a directory without -R turned on). VERSION
This man page is correct for version 3 of the Samba suite. AUTHOR
The original Samba software and related utilities were created by Andrew Tridgell. Samba is now developed by the Samba Team as an Open Source project similar to the way the Linux kernel is developed. The smbget manpage was written by Jelmer Vernooij. Samba 3.5 06/18/2010 SMBGET(1)
All times are GMT -4. The time now is 04:53 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy