How to use cURL to download web page with autentification (form)?

Login or Register to Reply

Thread Tools Search this Thread
# 1  
Old 08-04-2016
Question How to use cURL to download web page with autentification (form)?


I'm new in the forum and really beginer, and also sorry form my bad english.

I use linux and want to create little program to download automaticaly some pdf (invoices) and put in a folder of my computer. I learn how to do and programme with ubuntu but the program will be implemented on a mac osx so I look for compatible soft.

So I look for cURL.

I don't know if a subject is already open arround curl and authentification, if yes, please redirect me.

I can download a pdf from a command line (good begining) but the file I want to get are on a web site with authentification form, and it's impossible to go directly to the url of the pdf, I have to go to login page, and when it's done I fall in the "home" part of the web site with an other authentification.

So I have to pass the form, pass a second form with just a number "personal code", and after go to the url of the file I want to download.

I try to pass the first authentification but I dont know if it's done or not, and i dont know how to, after authentification, keep the cookies and continu to pass the second authentification and ask cURL for the url of the file.

To begin, some body can maybe say me if with this command line I pass the form with succes or not: (XXXXX is the pass word and username)

curl --trace-ascii debugcyclosoftware.txt -d klantid=XXXXX -d wachtwoord=XXXXX

You can see the form at the adress in the code.

And the debugcyclosoftware.txt is in attachement.

The second athentification is also a form but ask the "personnal code" to ride the web site.

When I look the debugcyclosoftware.txt file I see that I finish at location that I would like to go (home page), so it's done?

If i try to put a second url after the first to do a second post (for second form) I directly go back to the login page with a HTTP/1.1 302 Found.

But I maybe need a language like python or PHP to manage curl to do different action step by step and keep the cookie.

I don't know to mush php but I see that it work well with cURL.

Any advises or suggestion to help me to continu to learn and do?

I hope I m relatively clear...

Thanks a lot to having read =)
# 2  
Old 08-04-2016
I havent tried that with cURL, but used wget (a similar tool) to achieve about the same.

here is a test script i once wrote to write a bot for wiki-pages. It is not exactly a solution for you but shows how things work and how you can manipulate and use persistent login data across several calls of wget:

#! /bin/ksh93

typeset WGET=$(which wget)
typeset chWikiInst="my_wiki"
typeset chWikiURL="http://my_system/wiki/api.php"
typeset fWorkDir="/home/bakunin/projects/wiki/work"
typeset fOut="$fWorkDir/outfile"
typeset fTok="$fWorkDir/token"
typeset chUser="BotUser"
typeset chPwd="UserBot"
typeset chToken=""
typeset chSessionID=""
typeset chEditToken=""

rm "${fOut}*"
rm "${fTok}*"

# ----------------------- login1 --------------------
$WGET --post-data "action=login&lgname=${chUser}&lgpassword=${chPwd}&format=xml" \
      --save-cookies="${fTok}.login1" \
      --output-document="${fOut}.login1" \
      --keep-session-cookies \
      -q \

                                                       # extract info
chToken="$(sed 's/.*\ token="\([^"]*\)".*/\1/' "${fOut}.login1")"
chSessionID="$(sed 's/.*\ sessionid="\([^"]*\)".*/\1/' "${fOut}.login1")"

print - "sessionID: $chSessionID \t Token: $chToken"

# ----------------------- confirm token --------------------
$WGET --post-data "action=login&lgname=${chUser}&lgpassword=${chPwd}&lgtoken=${chToken}&format=xml" \
      --load-cookies="${fTok}.login1" \
      --save-cookies="${fTok}.login2" \
      --output-document="${fOut}.login2" \
      --keep-session-cookies \
      -q \

# ----------------------- get edit token --------------------
$WGET --post-data "action=tokens&type=edit&format=xml" \
      --load-cookies="${fTok}.login2" \
      --save-cookies="${fTok}.edit" \
      --output-document="${fOut}.edit" \
      --keep-session-cookies \
      -q \

                                                       # extract info
chEditToken="$(sed 's/.*\ edittoken="\([^"]*\)+\\".*/\1/' "${fOut}.edit")"
                                                       # pseudo-URL-encode trailing "+\"
print - "sessionID: $chSessionID\nToken....: $chToken\nEditToken: $chEditToken"

# ----------------------- create new page --------------------
$WGET --post-data "action=edit&title=MyTestPage&contentformat=text/x-wiki&format=xml&text='Hello-World'&token=${chEditToken}" \
      --load-cookies="${fTok}.edit" \
      --save-cookies="${fTok}.create" \
      --output-document="${fOut}.create" \
      --keep-session-cookies \
      -q \

# ----------------------- logout -------------------
$WGET --post-data "action=logout&format=xml" \
      --load-cookies="${fTok}.edit" \
      --save-cookies="${fTok}.sessionend" \
      --output-document="${fOut}.sessionend" \
exit 0

I hope this helps.

This User Gave Thanks to bakunin For This Post:
RavinderSingh13 (08-04-2016)
Login or Register to Reply

Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
CURL - Post Form Isssue ( sequel ) Fred13 Web Development 4 2 Weeks Ago 12:33 PM
Curl - post form issue Fred13 Web Development 3 2 Weeks Ago 06:11 AM
Curl login to web page Vit0_Corleone Linux 0 01-06-2017 09:36 PM
Contact Form for Download fedora18 Web Development 1 05-27-2015 06:19 PM
Curl Script - Post a file (multipart/form-data) Anooja G Shell Programming and Scripting 0 04-28-2014 09:08 AM
Use curl to send a static xml file using url encoding to a web page using pos Paul Walker Shell Programming and Scripting 1 11-26-2013 03:31 PM
List and download web page files shadyuk UNIX for Dummies Questions & Answers 2 07-18-2013 04:59 AM
Random web page download wget script shadyuk Shell Programming and Scripting 14 04-21-2013 05:21 AM
help pulling ${VARS} out of a web page user curl briandanielz Shell Programming and Scripting 1 08-09-2012 12:32 AM
Using cURL to submit a post form hansvg Shell Programming and Scripting 2 07-24-2012 03:50 PM
Help with using curl to download files from https henryN UNIX for Advanced & Expert Users 4 02-07-2012 06:50 PM
Download using curl and redirect question_please help santod Shell Programming and Scripting 1 03-07-2011 01:49 PM
Possible to download web page's text to a file? Breanne UNIX for Dummies Questions & Answers 1 02-16-2011 03:46 PM
Posting data to a form using curl DeCoTwc UNIX for Dummies Questions & Answers 0 05-30-2010 04:34 AM
maintaining form post session using curl Allasso Shell Programming and Scripting 1 01-13-2009 06:26 PM