Sponsored Content
Top Forums UNIX for Beginners Questions & Answers How to use cURL to download web page with authentification (form)? Post 302978812 by MarcelOrMittal on Thursday 4th of August 2016 08:18:00 AM
Old 08-04-2016
How to use cURL to download web page with authentification (form)?

Hello,

I'm new in the forum and really beginer, and also sorry form my bad english.

I use linux and want to create little program to download automaticaly some pdf (invoices) and put in a folder of my computer. I learn how to do and programme with ubuntu but the program will be implemented on a mac osx so I look for compatible soft.

So I look for cURL.

I don't know if a subject is already open arround curl and authentification, if yes, please redirect me.

So,
I can download a pdf from a command line (good begining) but the file I want to get are on a web site with authentification form, and it's impossible to go directly to the url of the pdf, I have to go to login page, and when it's done I fall in the "home" part of the web site with an other authentification.

So I have to pass the form, pass a second form with just a number "personal code", and after go to the url of the file I want to download.

I try to pass the first authentification but I dont know if it's done or not, and i dont know how to, after authentification, keep the cookies and continu to pass the second authentification and ask cURL for the url of the file.

To begin, some body can maybe say me if with this command line I pass the form with succes or not: (XXXXX is the pass word and username)

Code:
curl --trace-ascii debugcyclosoftware.txt -d klantid=XXXXX -d wachtwoord=XXXXX https://s02.cyclesoftware.nl/app/cs/account/login/login/

You can see the form at the adress in the code.

And the debugcyclosoftware.txt is in attachement.

The second athentification is also a form but ask the "personnal code" to ride the web site.

When I look the debugcyclosoftware.txt file I see that I finish at location that I would like to go (home page), so it's done?

If i try to put a second url after the first to do a second post (for second form) I directly go back to the login page with a HTTP/1.1 302 Found.

But I maybe need a language like python or PHP to manage curl to do different action step by step and keep the cookie.

I don't know to mush php but I see that it work well with cURL.

Any advises or suggestion to help me to continu to learn and do?

I hope I m relatively clear...

Thanks a lot to having read =)
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Posting data to a form using curl

Hello all. I have an incredible number of servers that I need to change a parameter on using a web interface. I'd like to be able to do this via curl, but I'm having some trouble. I filled out the form and hit update while snooping (tcpdump) my interface. That gave the the following as what is... (0 Replies)
Discussion started by: DeCoTwc
0 Replies

2. UNIX for Dummies Questions & Answers

Possible to download web page's text to a file?

Hi, Say there is a web page that contains just text only - that is, even the source code is just the text itself, nothing more. An example would be "http://mynasadata.larc.nasa.gov/docs/ocean_percent.txt" Is there a UNIX command that would allow me to download this text and store it in a... (1 Reply)
Discussion started by: Breanne
1 Replies

3. Shell Programming and Scripting

Using cURL to submit a post form

I am trying to write a shell script to use curl in order to automate downloading data from a website. The URL with the post form is here: http://try-db.org/de/InfoBySpecies.php . I have a list of about 1800 different species I want to check. For Example, choose the first species and use the... (2 Replies)
Discussion started by: hansvg
2 Replies

4. Shell Programming and Scripting

help pulling ${VARS} out of a web page user curl

Here is the code I have so far #!/bin/bash INFOF="/tmp/mac.info" curl --silent http://www.everymac.com/systems/apple/macbook_pro/specs/macbook-pro-core-2-duo-2.8-aluminum-17-mid-2009-unibody-specs.html "$INFOF" I want help putting these specs into a vars Standard Ram: value into $VAR1... (1 Reply)
Discussion started by: briandanielz
1 Replies

5. Solaris

Disable SNMP v1/2 authentification and use SNMPv3 authentification

Hello everyone, Can anyone help me to disable SNMP v1/2 authentification on Solaris 10 and use SNMP v3 authentification ? I have SNMPv3 installed on my Solaris but the tool we use to scan vulnerabilities still show that there is a problem whith authentification version. Does it mean that... (1 Reply)
Discussion started by: adilyos
1 Replies

6. Shell Programming and Scripting

Random web page download wget script

Hi, I've been attempting to create a script that downloads web pages at random intervals to mimic typical user usage. However I'm struggling to link $url to the URL list and thus wget complains of a missing URL. Any ideas? Thanks #!/bin/sh #URL List url1="http://www.bbc.co.uk"... (14 Replies)
Discussion started by: shadyuk
14 Replies

7. UNIX for Dummies Questions & Answers

List and download web page files

Hello, Does anyone know of a way to list all files related to a single web page and then to download say 4 files at a time simultaneously until the entire web page has been downloaded successfully? I'm essentially trying to mimic a browser. Thanks. (2 Replies)
Discussion started by: shadyuk
2 Replies

8. Shell Programming and Scripting

Use curl to send a static xml file using url encoding to a web page using pos

Hi I am try to use curl to send a static xml file using url encoding to a web page using post. This has to go through a particular port on our firewall as well. This is my first exposure to curl and am not having much success, so any help you can supply, or point me in the right direction would be... (1 Reply)
Discussion started by: Paul Walker
1 Replies

9. Linux

Curl login to web page

Hello dears, I am trying to log in the website using curl but no luck so far. The web page Content-Type is : text/html;charset=ISO-8859-1 I try the following command using curl: curl \ --header "Content-type: text/html" \ --request POST \ --data '{"user": "someusername",... (0 Replies)
Discussion started by: Vit0_Corleone
0 Replies

10. Web Development

Curl - post form issue

I'm having an issue with curl post form, I dont' understand what I'm mising. I would like to send a post command login/password to a form, quite simple in the paper. URL : http: // <myebsite> / login Here the form source code : <form action="/login_check" method="post"> <input... (3 Replies)
Discussion started by: Fred13
3 Replies
AXEL(1) 						      General Commands Manual							   AXEL(1)

NAME
Axel - A light download accelerator for Linux. SYNOPSIS
axel [OPTIONS] url1 [url2] [url...] DESCRIPTION
Axel is a program that downloads a file from a FTP or HTTP server through multiple connection, each connection downloads its own part of the file. Unlike most other programs, Axel downloads all the data directly to the destination file, using one single thread. It just saves some time at the end because the program doesn't have to concatenate all the downloaded parts. OPTIONS
One argument is required, the URL to the file you want to download. When downloading from FTP, the filename may contain wildcards and the program will try to resolve the full filename. Multiple URL's can be specified as well and the program will use all those URL's for the download. Please note that the program does not check whether the files are equal. Other options: --max-speed=x, -s x You can specify a speed (bytes per second) here and Axel will try to keep the average speed around this speed. Useful if you don't want the program to suck up all of your bandwidth. --num-connections=x, -n x You can specify an alternative number of connections here. --output=x, -o x Downloaded data will be put in a local file with the same name, unless you specify a different name using this option. You can spec- ify a directory as well, the program will append the filename. --search[=x], -S[x] Axel can do a search for mirrors using the filesearching.com search engine. This search will be done if you use this option. You can specify how many different mirrors should be used for the download as well. The search for mirrors can be time-consuming because the program tests every server's speed, and it checks whether the file's still available. --no-proxy, -N Don't use any proxy server to download the file. Not possible when a transparent proxy is active somewhere, of course. --verbose If you want to see more status messages, you can use this option. Use it more than once if you want to see more. --quiet, -q No output to stdout. --alternate, -a This will show an alternate progress indicator. A bar displays the progress and status of the different threads, along with current speed and an estimate for the remaining download time. --header=x, -H x Add an additional HTTP header. This option should be in the form "Header: Value". See RFC 2616 section 4.2 and 14 for details on the format and standardized headers. --user-agent=x, -U x Set the HTTP user agent to use. Some websites serve different content based upon this parameter. The default value will include "Axel", its version and the platform. --help, -h A brief summary of all the options. --version, -V Get version information. NOTE
Long (double dash) options are supported only if your platform knows about the getopt_long call. If it does not (like *BSD), only the short options can be used. RETURN VALUE
The program returns 0 when the download was succesful, 1 if something really went wrong and 2 if the download was interrupted. If something else comes back, it must be a bug.. EXAMPLES
axel ftp://ftp.{be,nl,uk,de}.kernel.org/pub/linux/kernel/v2.4/linux-2.4.17.tar.bz2 This will use the Belgian, Dutch, English and German kernel.org mirrors to download a Linux 2.4.17 kernel image. axel -S4 ftp://ftp.kernel.org/pub/linux/kernel/v2.4/linux-2.4.17.tar.bz2 This will do a search for the linux-2.4.17.tar.bz2 file on filesearching.com and it'll use the four (if possible) fastest mirrors for the download. (Possibly including ftp.kernel.org) (Of course, the commands are a single line, but they're too long to fit on one line in this page.) FILES
/etc/axelrc System-wide configuration file. Note that development versions place this file in /usr/local/etc. ~/.axelrc Personal configuration file These files are not documented in a man-page, but the example file which comes with the program contains enough information, I hope. The position of the system-wide configuration file might be different. COPYRIGHT
Axel is Copyright 2001-2002 Wilmer van der Gaast. BUGS
Please report bugs at https://alioth.debian.org/tracker/?group_id=100070&atid=413085. AUTHORS
Wilmer van der Gaast. <wilmer@gaast.net> AXEL(1)
All times are GMT -4. The time now is 08:03 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy