Using WGET or CURL to HTTP Post with Authentication | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Using WGET or CURL to HTTP Post with Authentication

Shell Programming and Scripting


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 02-26-2010
krypton krypton is offline
Registered User
 
Join Date: Feb 2010
Last Activity: 5 January 2012, 11:07 AM EST
Posts: 7
Thanks: 4
Thanked 0 Times in 0 Posts
Using WGET or CURL to HTTP Post with Authentication

Hello All,

I am currently trying to use either WGET or CURL to automate some HTTP functions that otherwise required a lot of repeated manual steps.

From looking at the MAN pages from CURL and WGET there is some basic information concerning HTTP POST but nothing which explains exactly which steps would be needed to include a full update.

Looking at a packet trace from a POST the following content is usually sent to the webserver to get the achieved results:


Code:
POST /XRDTool/update/Update.do HTTP/1.1
Host: <snip>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10
Content-Type: application/x-www-form-urlencoded
Referer: http://<snip>
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Origin: <snip>
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Cookie: JSESSIONID=<snip>6; maintab=tcontent1; CP_GUTC=<snip>
Content-Length: 699
Connection: keep-alive

Number=<truncated>

With only a few options needing to be changed from one post to the next.

Has anybody had experience with automating such a function in the past or could suggest how to go about performing this?

Best Regards,

krypton

Last edited by pludi; 02-26-2010 at 09:21 AM.. Reason: added code tags
Sponsored Links
    #2  
Old 02-26-2010
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 18 October 2014, 9:20 PM EDT
Location: Saskatchewan
Posts: 19,614
Thanks: 811
Thanked 3,329 Times in 3,118 Posts
You need to send a cookie to authenticate these pages it seems. You also have to get the cookie in the first place. Check out wget's --save-cookies, --load-cookies, and --keep-session-cookies settings.

From wget's manual page:

Code:
		   # Log in to the server.  This can be done only once.
		   wget --save-cookies cookies.txt \
			--post-data 'user=foo&password=bar' \
			http://server.com/auth.php

		   # Now grab the page or pages we care about.
		   wget --load-cookies cookies.txt \
			-p http://server.com/interesting/article.php

Sponsored Links
    #3  
Old 02-26-2010
fubaya fubaya is offline
Registered User
 
Join Date: Oct 2009
Last Activity: 3 September 2011, 11:25 PM EDT
Posts: 74
Thanks: 2
Thanked 3 Times in 3 Posts
Here is an example for using curl to post data (may need the -c option to use cookies). The formfind.pl script I linked to in that post makes it much easier to find out what information has to be posted to log in with curl or wget.
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Unable to access http site using wget through proxy memonks Shell Programming and Scripting 2 06-21-2009 01:32 AM
piping a url to wget or curl? ropers Shell Programming and Scripting 5 03-21-2009 12:35 AM
cant make a http get request using wget elthox UNIX for Dummies Questions & Answers 1 02-20-2009 01:49 PM
wget using wildcards using http hcclnoodles Shell Programming and Scripting 2 04-13-2005 10:50 AM



All times are GMT -4. The time now is 12:27 AM.