Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Using Wget with Digest Authentication

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using Wget with Digest Authentication
# 1  
Old 06-26-2012
Using Wget with Digest Authentication
I’m trying to use wget 1.13.4 from a command line to access a web page using Digest Authentication on a Tomcat server, and I keep getting HTTP/1.1 401 Unauthorized response. Below is my code and the response. Any suggestions would be appreciated.

wget --debug --server-response --user-agent="" --ignore-case --user=admin --password=abc -O - <URL_here>
Setting --server-response (serverresponse) to 1
Setting --user-agent (useragent) to
Setting --ignore-case (ignorecase) to 1
Setting --user (user) to admin
Setting --password (password) to abc
Setting --output-document (outputdocument) to -
DEBUG output created by Wget 1.13.4 on linux-gnu.

--2012-06-26 07:58:14--
<URL_here>
Host `10.107.70.21' has not issued a general basic challenge.
Connecting to 10.107.70.21:8080... connected.
Created socket 3.
Releasing 0x000000001174a680 (new refcount 0).
Deleting unused 0x000000001174a680.

---request begin---
GET /admin/console/View.jsp HTTP/1.1
Accept: */*
Host: 10.107.70.21:8080
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1
Pragma: No-cache
Cache-Control: no-cache
Expires: Wed, 31 Dec 1969 19:00:00 EST
WWW-Authenticate: Digest realm="Admin Console", qop="auth", nonce="c53281ac78f4ad7dcf903d29372db789", opaque="1340c4ba6a13a14 d0d0bcc6262d8ad83"
Content-Type: text/html;charset=utf-8
Content-Length: 954
Date: Tue, 26 Jun 2012 11:58:14 GMT

---response end---

HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1
Pragma: No-cache
Cache-Control: no-cache
Expires: Wed, 31 Dec 1969 19:00:00 EST
WWW-Authenticate: Digest realm="Admin Console", qop="auth", nonce="c53281ac78f4ad7dcf903d29372db789", opaque="1340c4ba6a13a 14d0d0bcc6262d8ad83"
Content-Type: text/html;charset=utf-8
Content-Length: 954
Date: Tue, 26 Jun 2012 11:58:14 GMT
Registered socket 3 for persistent reuse.
Skipping 954 bytes of body: [<html><head><title>Apache Tomcat/6.0.29 - Error report</title><style><!--H1 {font-family:Tahoma,Arial ,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-c olor:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY { font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;backgr ound-color:#525D76;} Skipping 442 bytes of body: [P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12 px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 401 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>This request requires HTT P authentication ().</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.29</h3></body></html>] done.
Reusing existing connection to 10.107.70.21:8080.
Reusing fd 3.

---request begin---
GET /admin/console/View.jsp HTTP/1.1
Accept: */*
Host: 10.107.70.21:8080
Connection: Keep-Alive
Authorization: Digest username="admin", realm="Admin Console", nonce="c53281ac78f4ad7dcf903d29372db789", uri="/admin/console/ View.jsp", response="7cd7f9b4271d4e2bfb18a112ab9a42d0", opaque="1340c4ba6a13a14d0d0bcc6262d8ad83"

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1
Pragma: No-cache
Cache-Control: no-cache
Expires: Wed, 31 Dec 1969 19:00:00 EST
WWW-Authenticate: Digest realm="Admin Console", qop="auth", nonce="b0da6404ba288764a129deb4b80b2f9e", opaque="62b6cc5282e54f6 da2b94814ded090f6"
Content-Type: text/html;charset=utf-8
Content-Length: 954
Date: Tue, 26 Jun 2012 11:58:14 GMT

---response end---

HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1
Pragma: No-cache
Cache-Control: no-cache
Expires: Wed, 31 Dec 1969 19:00:00 EST
WWW-Authenticate: Digest realm="Admin Console", qop="auth", nonce="b0da6404ba288764a129deb4b80b2f9e", opaque="62b6cc5282e54 f6da2b94814ded090f6"
Content-Type: text/html;charset=utf-8
Content-Length: 954
Date: Tue, 26 Jun 2012 11:58:14 GMT
Skipping 954 bytes of body: [<html><head><title>Apache Tomcat/6.0.29 - Error report</title><style><!--H1 {font-family:Tahoma,Arial ,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-c olor:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY { font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;backgr ound-color:#525D76;} Skipping 442 bytes of body: [P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12 px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 401 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>This request requires HTT P authentication ().</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.29</h3></body></html>] done.
Authorization failed.
Last edited by vbe; 06-26-2012 at 10:59 AM..
# 2  
Old 06-28-2012
I would like to thank everyone who has responded, a lot of good information was provided but unfortunately I still cannot connect with Wget and get the same error response. The Tomcat server has othe Basic Auth pages and I can use Wget successfully with them but not the 1 page using digest auth. Here is what I have tried without success.
- confirmed ID and password work from a browser.
- tried wrapping the ID and password in single quotes.
- tried using the --header option
- installed the latest version to eliminate the bug in earlier versions
- tried using curl -w -S -v -u admin:abc --digest <URL_here>

Thanks to Tim R., Wget is now working for the version of Digest Authentication on my server.

What Tim discovered:
Your server expects the client to use RFC 2617 Digest Access Authentication.
Wget just supports RFC 2069 Digest Access Authentication - i took a look into wget's source code.
Firefox - and maybe all other modern browsers - uses (of course) RFC 2617.

That's why Firefox works ok and wget doesn't.

To solve your problem, you can either
- configure your server to accept RFC 2069 (which is a bit unsafer)
- wait until someone extends wget (which seems not to be a big deal)

Then I received the below patch and ran it in on file http.c in directory: wget/wget-1.13.4/src/
Patch -p1 < 0001-add-support-for-RFC-2617-Digest-Access-Authenticatio.patch
cd ../
sudo make install
Wget now is able to connect to the Digest protected page.

Thanks to everyone for their help on this.

Hereis the code for the patch.

From 649b8693d699c28830fc60f5da2c11ae83fdb22b Mon Sep 17 00:00:00 2001
From: Tim R.
Date: Thu, 28 Jun 2012 17:45:18 +0200
Subject: [PATCH] * add support for RFC 2617 Digest Access Authentication
---
src/http.c | 98 ++++++++++++++++++++++++++++++++++++++++++++++--------------
1 file changed, 76 insertions(+), 22 deletions(-)
diff --git a/src/http.c b/src/http.c
index 8d4edba..9ff7f28 100644
--- a/src/http.c
+++ b/src/http.c
@@ -3655,19 +3655,23 @@ digest_authentication_encode (const char *au, const char *user,
const char *passwd, const char *method,
const char *path)
{
- static char *realm, *opaque, *nonce;
+ static char *realm, *opaque, *nonce, *qop;
static struct {
const char *name;
char **variable;
} options[] = {
{ "realm", &realm },
{ "opaque", &opaque },
- { "nonce", &nonce }
+ { "nonce", &nonce },
+ { "qop", &qop }
};
+ char cnonce[16] = "";
char *res;
+ size_t res_size;
param_token name, value;

- realm = opaque = nonce = NULL;
+
+ realm = opaque = nonce = qop = NULL;

au += 6; /* skip over `Digest' */
while (extract_param (&au, &name, &value, ','))
@@ -3683,11 +3687,19 @@ digest_authentication_encode (const char *au, const char *user,
break;
}
}
+
+ if (qop!=NULL && strcmp(qop,"auth"))
+ {
+ logprintf (LOG_NOTQUIET, _("Unsupported quality of protection '%s'.\n"), qop);
+ user=NULL; /* force freeing mem and return */
+ }
+
if (!realm || !nonce || !user || !passwd || !path || !method)
{
xfree_null (realm);
xfree_null (opaque);
xfree_null (nonce);
+ xfree_null (qop);
return NULL;
}

@@ -3716,27 +3728,69 @@ digest_authentication_encode (const char *au, const char *user,
md5_finish_ctx (&ctx, hash);
dump_hash (a2buf, hash);

- /* RESPONSE_DIGEST = H(A1BUF ":" nonce ":" A2BUF) */
- md5_init_ctx (&ctx);
- md5_process_bytes ((unsigned char *)a1buf, MD5_DIGEST_SIZE * 2, &ctx);
- md5_process_bytes ((unsigned char *)":", 1, &ctx);
- md5_process_bytes ((unsigned char *)nonce, strlen (nonce), &ctx);
- md5_process_bytes ((unsigned char *)":", 1, &ctx);
- md5_process_bytes ((unsigned char *)a2buf, MD5_DIGEST_SIZE * 2, &ctx);
- md5_finish_ctx (&ctx, hash);
+ if (!strcmp(qop,"auth"))
+ {
+ /* RFC 2617 Digest Access Authentication */
+ /* generate random hex string */
+ snprintf(cnonce, sizeof(cnonce), "%08x", random_number(INT_MAX));
+
+ /* RESPONSE_DIGEST = H(A1BUF ":" nonce ":" noncecount ":" clientnonce ":" qop ": " A2BUF) */
+ md5_init_ctx (&ctx);
+ md5_process_bytes ((unsigned char *)a1buf, MD5_DIGEST_SIZE * 2, &ctx);
+ md5_process_bytes ((unsigned char *)":", 1, &ctx);
+ md5_process_bytes ((unsigned char *)nonce, strlen (nonce), &ctx);
+ md5_process_bytes ((unsigned char *)":", 1, &ctx);
+ md5_process_bytes ((unsigned char *)"00000001", 8, &ctx); /* TODO: keep track of server nonce values */
+ md5_process_bytes ((unsigned char *)":", 1, &ctx);
+ md5_process_bytes ((unsigned char *)cnonce, strlen(cnonce), &ctx);
+ md5_process_bytes ((unsigned char *)":", 1, &ctx);
+ md5_process_bytes ((unsigned char *)qop, strlen(qop), &ctx);
+ md5_process_bytes ((unsigned char *)":", 1, &ctx);
+ md5_process_bytes ((unsigned char *)a2buf, MD5_DIGEST_SIZE * 2, &ctx);
+ md5_finish_ctx (&ctx, hash);
+ }
+ else
+ {
+ /* RFC 2069 Digest Access Authentication */
+ /* RESPONSE_DIGEST = H(A1BUF ":" nonce ":" A2BUF) */
+ md5_init_ctx (&ctx);
+ md5_process_bytes ((unsigned char *)a1buf, MD5_DIGEST_SIZE * 2, &ctx);
+ md5_process_bytes ((unsigned char *)":", 1, &ctx);
+ md5_process_bytes ((unsigned char *)nonce, strlen (nonce), &ctx);
+ md5_process_bytes ((unsigned char *)":", 1, &ctx);
+ md5_process_bytes ((unsigned char *)a2buf, MD5_DIGEST_SIZE * 2, &ctx);
+ md5_finish_ctx (&ctx, hash);
+ }
+
dump_hash (response_digest, hash);

- res = xmalloc (strlen (user)
- + strlen (user)
- + strlen (realm)
- + strlen (nonce)
- + strlen (path)
- + 2 * MD5_DIGEST_SIZE /*strlen (response_digest)*/
- + (opaque ? strlen (opaque) : 0)
- + 128);
- sprintf (res, "Digest \
-username=\"%s\", realm=\"%s\", nonce=\"%s\", uri=\"%s\", response=\"%s\"",
- user, realm, nonce, path, response_digest);
+ res_size = strlen (user)
+ + strlen (user)
+ + strlen (realm)
+ + strlen (nonce)
+ + strlen (path)
+ + 2 * MD5_DIGEST_SIZE /*strlen (response_digest)*/
+ + (opaque ? strlen (opaque) : 0)
+ + (qop ? 128: 0)
+ + 128;
+
+ res = xmalloc (res_size);
+
+ if (!strcmp(qop,"auth"))
+ {
+ snprintf (res, res_size, "Digest "\
+ "username=\"%s\", realm=\"%s\", nonce=\"%s\", uri=\"%s\", response=\"%s\""\
+ ", qop=auth, nc=00000001, cnonce=\"%s\"",
+ user, realm, nonce, path, response_digest, cnonce);
+
+ }
+ else
+ {
+ snprintf (res, res_size, "Digest "\
+ "username=\"%s\", realm=\"%s\", nonce=\"%s\", uri=\"%s\", response=\"%s\"",
+ user, realm, nonce, path, response_digest);
+ }
+
if (opaque)
{
char *p = res + strlen (res);
--
1.7.10
Last edited by mtarkowski; 06-28-2012 at 04:56 PM.. Reason: Problem Solved
These 2 Users Gave Thanks to mtarkowski For This Post:
alister Corona688
# 3  
Old 06-28-2012
Quote:
Originally Posted by mtarkowski
I would like to thank everyone who has responded, a lot of good information was provided
Here's some more good information for you: Your sarcasm makes for a poor first impression. We are not your personal, paid technical support team.

Regards,
Alister
# 4  
Old 06-28-2012
No sarcasm was intended with my comment. If I offended you in any way I do apologize.
# 5  
Old 06-28-2012
I'm guessing it's a language barrier. 'everyone who tried' would have made a lot more sense. I took a look at it but couldn't figure out the problem.

Thank you for posting your solution. That is very helpful to know.
# 6  
Old 06-28-2012
Quote:
Originally Posted by mtarkowski
No sarcasm was intended with my comment. If I offended you in any way I do apologize.
At the time, it seemed like a snarky comment regarding the lack of responses. In light of the information you added to the post, it now seems you were simply referring to help you had received elsewhere. Sorry about that.

Thank you very much for following up and providing detailed information regarding the cause of the problem and the patch. I, for one, wasn't aware of this issue.

Welcome to the forum.

Regards,
Alister
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403... (2 Replies)
Discussion started by: Prasannag87
2 Replies

2. Shell Programming and Scripting

Wget and gz

Can wget be used to goto a site and piped into a .gz extrated command? wget ftp://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh37 | gunzip -d clinvar_20150603.vcf.gz (1 Reply)
Discussion started by: cmccabe
1 Replies

3. Red Hat

Digest::md5 needed for RHEL 6.4

Hey, I've got a package that requires 'Digest:md5' to be installed but I can't locate a source for it from Redhat. Poking around on the internet I found some stuff that seemed to be for RHEL 6.4 but when I had yum try a local install it gives me this error. Transaction Check Error: file... (3 Replies)
Discussion started by: DustinT
3 Replies

4. Red Hat

Wget

If I run the following command wget -r --no-parent --reject "index.html*" 10.11.12.13/backups/ A local directory named 10.11.12.13/backups with the content of web site data is created. What I want to do is have the data placed in a local directory called $HOME/backups. Thanks for... (1 Reply)
Discussion started by: popeye
1 Replies

5. Shell Programming and Scripting

WGET help!

Hi Friends, I have an url like this https://www.unix.com/help/ In this help directory, I have more than 300 directories which contains file or files. So, the 300 directories are like this http://unix.com/help/ dir1 file1 dir2 file2 dir3 file3_1 file3_2... (4 Replies)
Discussion started by: jacobs.smith
4 Replies

6. UNIX for Dummies Questions & Answers

Wget

...... (1 Reply)
Discussion started by: hoo
1 Replies

7. UNIX for Dummies Questions & Answers

Different MD5 value when using 'digest' command

Hi guys, I need to anonymise some data; Some of it in an Oracle database and some in text files. I'm using the 'digest' command on Solaris 10 OS and an Oracle function to encode the data in the database. However, as a test, when i try to encode the same string in the dataabse ans OS, I get... (4 Replies)
Discussion started by: zaff
4 Replies

8. Shell Programming and Scripting

wget

Hi I want to download some files using wget , and want to save in a specified directory. Is there any way to save it.Please suggest me. (1 Reply)
Discussion started by: mnmonu
1 Replies

9. Shell Programming and Scripting

Help with wget

Hi, i need temperature hourly from a web page Im using wget to get the web page. I would like to save the page downloaded in a file called page. I check the file everytime i run the wget function but its not saving but instead creates a wx.php file....Each time i run it...a new wx.php file is... (2 Replies)
Discussion started by: vadharah
2 Replies

10. UNIX for Advanced & Expert Users

Digest MD5

Dear Guys , Am sorry i ask alot , but i do not know that much about perl , cgi , MD5 ! now i installed MD5 and Digest MD5 to my solaries 7 sparc machine . when i execute the command : $perl Makefile.PL i get the follwoing error message ,, please tell me how to fix it , i need... (11 Replies)
Discussion started by: tamemi
11 Replies
Login or Register to Ask a Question