Unix/Linux Go Back    

Shell Programming and Scripting Unix shell scripting - KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and shell scripts and shell scripting languages here.

Using WGET or CURL to HTTP Post with Authentication

Shell Programming and Scripting

Closed Linux or Unix Question    
Thread Tools Search this Thread Display Modes
Old Unix and Linux 02-26-2010
krypton krypton is offline
Registered User
Join Date: Feb 2010
Last Activity: 5 January 2012, 11:07 AM EST
Posts: 7
Thanks: 4
Thanked 0 Times in 0 Posts
Using WGET or CURL to HTTP Post with Authentication

Hello All,

I am currently trying to use either WGET or CURL to automate some HTTP functions that otherwise required a lot of repeated manual steps.

From looking at the MAN pages from CURL and WGET there is some basic information concerning HTTP POST but nothing which explains exactly which steps would be needed to include a full update.

Looking at a packet trace from a POST the following content is usually sent to the webserver to get the achieved results:

POST /XRDTool/update/Update.do HTTP/1.1
Host: <snip>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10
Content-Type: application/x-www-form-urlencoded
Referer: http://<snip>
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Origin: <snip>
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Cookie: JSESSIONID=<snip>6; maintab=tcontent1; CP_GUTC=<snip>
Content-Length: 699
Connection: keep-alive


With only a few options needing to be changed from one post to the next.

Has anybody had experience with automating such a function in the past or could suggest how to go about performing this?

Best Regards,


Last edited by pludi; 02-26-2010 at 10:21 AM.. Reason: added code tags
Sponsored Links
Old Unix and Linux 02-26-2010
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
Join Date: Aug 2005
Last Activity: 4 February 2016, 3:16 PM EST
Location: Saskatchewan
Posts: 21,181
Thanks: 938
Thanked 3,813 Times in 3,557 Posts
You need to send a cookie to authenticate these pages it seems. You also have to get the cookie in the first place. Check out wget's --save-cookies, --load-cookies, and --keep-session-cookies settings.

From wget's manual page:

		   # Log in to the server.  This can be done only once.
		   wget --save-cookies cookies.txt \
			--post-data 'user=foo&password=bar' \

		   # Now grab the page or pages we care about.
		   wget --load-cookies cookies.txt \
			-p http://server.com/interesting/article.php

Sponsored Links
Old Unix and Linux 02-26-2010
fubaya fubaya is offline
Registered User
Join Date: Oct 2009
Last Activity: 3 September 2011, 11:25 PM EDT
Posts: 74
Thanks: 2
Thanked 3 Times in 3 Posts
Here is an example for using curl to post data (may need the -c option to use cookies). The formfind.pl script I linked to in that post makes it much easier to find out what information has to be posted to log in with curl or wget.
Sponsored Links
Closed Linux or Unix Question

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Unable to access http site using wget through proxy memonks Shell Programming and Scripting 2 06-21-2009 02:32 AM
piping a url to wget or curl? ropers Shell Programming and Scripting 5 03-21-2009 01:35 AM
cant make a http get request using wget elthox UNIX for Dummies Questions & Answers 1 02-20-2009 02:49 PM
wget using wildcards using http hcclnoodles Shell Programming and Scripting 2 04-13-2005 11:50 AM

All times are GMT -4. The time now is 07:53 PM.