Sponsored Content
Top Forums Shell Programming and Scripting Shell Script for Upload/download files using cURL Post 302484363 by dufftime on Thursday 30th of December 2010 06:38:27 PM
Old 12-30-2010
Sounds like you need HTTP POST's

I recently wrote a script to simulate some web transactions recently (not too different than yours), I had good luck following this tip from curl's website (hxxp://curl.haxx.se/docs/httpscripting.html):

Quote:
4.5 Figure Out What A POST Looks Like

When you're about fill in a form and send to a server by using curl instead
of a browser, you're of course very interested in sending a POST exactly the
way your browser does.

An easy way to get to see this, is to save the HTML page with the form on
your local disk, modify the 'method' to a GET, and press the submit button
(you could also change the action URL if you want to).

You will then clearly see the data get appended to the URL, separated with a
'?'-letter as GET forms are supposed to.
If there are 3 clicks, then there are going to be 3 separate curl commands. You should be able to avoid the first "click", because you'll just be uploading the file in step 2, no need to browse to the first page once you figure out the URL of your POST.

I'm making up examples in as I don't know what your actual HTML form looks like. The trick is figuring out the form query variables (order matters!!) and converting that into your curl command-line arguments.

Code:
#/bin/sh
# 1.  Get page - probably not required
curl http://www.mysite.com/mypage.htm

# 2.  Upload file - most likely a multipart form.  the '@' means upload a local file
curl -F "upload_filename=@/full/path/to/local/file.txt" -F "myotherqueryvar=queryvalue" \
-L http://www.mysite.com/upload-page.htm > html_out.htm

# 3.  Parse HTML - you'll basically have to look at the return page
# and see if you can figure out a pattern that will grab the link.
mylink=`grep (insert pattern here that will get your link) html_out.htm`
curl ${mylink}

Not sure if you can just supply the actual website... might be easier to give you the exact commands.

-dufftime
This User Gave Thanks to dufftime For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Getting upload and download speeds with the help of script?How?

How can I get the upload and download speed of my pc, provided that my interface is wlan0? I have though something like: #!/bin/sh ups=something downs=somethingelse echo " Your current up speed is $ups Your current down speed is $downs, but I have no idea how to get them?Are... (5 Replies)
Discussion started by: hakermania
5 Replies

2. Shell Programming and Scripting

Shell Script to monitor folder and upload found files via FTP

Hi everyone! I'm in a need of a shell script that search for all files in a folder, move all those files to a temp folder, and upload those files via FTP. When the file transfer via FTP completes successfully, the file is moved to a completed folder. In case any of those files fails, the file... (4 Replies)
Discussion started by: pulsorock
4 Replies

3. Shell Programming and Scripting

curl script to download files from Secured HTTPS server?

curl -# -v -d "sendusername=myname&password=mypassword&wheretogo=download.php" -L -o test.zip http://www.ims-dm.com/cgi/securedownload.php?p=HIREFTPM\&prodtype=hire/test.zip * About to connect() to www.ims-dm.com port 80 * Trying 209.61.193.139... connected * Connected to www.ims-dm.com... (1 Reply)
Discussion started by: laknar
1 Replies

4. UNIX for Dummies Questions & Answers

Shell Imgur upload with curl

Hi imgur(){ $*|convert label:@- png:-|curl -F "image=@-" -F "key=1913b4ac473c692372d108209958fd15" http://api.imgur.com/2/upload.xml|grep -Eo "<original>(.)*</original>" | grep -Eo "http://i.imgur.com/*";}Execute a command, convert output to .png file, upload file to imgur.com, then returning... (10 Replies)
Discussion started by: slashdotweenie
10 Replies

5. UNIX for Advanced & Expert Users

Help with using curl to download files from https

Hi I'm trying to download an xml file from a https server using curl on a Linux machine with Ubuntu 10.4.2 I am able to connect to the remote server with my username and password but the output is only "Virtual user <username> logged in". I am expecting to download the xml file. My output... (4 Replies)
Discussion started by: henryN
4 Replies

6. Shell Programming and Scripting

Curl ftp ssl download files

Hello all, I have been struggling with this issue on and off for a couple of weeks now and I just got it all working, so I wanted to share my findings in case some other poor soul needs to know how. First some background on what I'm doing. I am uploading files to different directories based on... (0 Replies)
Discussion started by: msjkadams
0 Replies

7. UNIX for Dummies Questions & Answers

How to download files matching pattern from FTP using CURL or WGET?

Hi, For an order I requested, the provider has uploaded a tar file in public FTP site which internally has tons of files (compressed) and I need to download files that follows particular pattern which would be few hundreds. Note: The order can't be requested for files that follows the... (7 Replies)
Discussion started by: Amalan
7 Replies

8. Shell Programming and Scripting

FTP in shell script and selecting files for upload

Hi, Im a newbie with programming and shell scripting. Im running OSX/Mac and Darwin. I would like to create a shell script that would : 1. Search a Volume and directory (including subdirectories) for a file that : * filename ends with ”_Highres.pdf” and * the file creation date of... (8 Replies)
Discussion started by: NickeZ28
8 Replies

9. Shell Programming and Scripting

Curl command to download multiple files with a file prefix

I am using the below curl command to download a single file from client server and it is working as expected curl --ftp-ssl -k -u ${USER}:${PASSWD} ftp://${HOST}:${PORT}/path/to/${FILE} --output ${DEST}/${FILE} let say the client has 3 files hellofile.101, hellofile.102, hellofile.103 and I... (3 Replies)
Discussion started by: r@v!7*7@
3 Replies

10. Shell Programming and Scripting

Curl , download file with user:pass in bash script

Hello, My question is about curl command. (ubuntu14.04) In terminal, I am able to download my mainfile with: curl -u user1:pass1 http://11.22.33.44/******* When I convert it into bash script like this: #!/bin/bash cd /root/scripts computer_ip=11.22.33.44 curl -u $1:$2... (8 Replies)
Discussion started by: baris35
8 Replies
DJVUSERVE(1)							   DjVuLibre-3.5						      DJVUSERVE(1)

NAME
djvuserve - Generate indirect DjVu documents on the fly. DESCRIPTION
Program djvuserve is a CGI program that can be executed by a HTTP server for serving DjVu documents. This program is able to convert a bundled multi-page document into an indirect document on the fly. USING DJVUSERVE
Program djvuserve must first be installed as a CGI program for your web server. There are several ways to achieve this. The Apache web server, for instance, often defines a specific directory for CGI programs using the ScriptAlias directive. Assume that the file httpd.conf contains the following line: ScriptAlias /cgi-bin/ "/var/www/cgi-bin" It is then sufficient to create a small executable shell script /var/www/cgi-bin/djvuserve containing the following lines: #!/bin/sh exec /full/path/to/djvuserve Suppose that a large bundled multi-page DjVu document is available at the following URL. http://server/dir/doc.djvu The CGI program djvuserve lets you access this same document as an indirect multi-page DjVu document using the following URL. http://server/cgi-bin/djvuserve/dir/doc.djvu/index.djvu Serving indirect multi-page DjVu documents provides for efficiently browsing large document without transferring unnecessary pages over the network. See djvu(1) for more information. Furthermore djvuserve searches certain keywords among the CGI arguments of the URL. The keyword bundled forces serving a bundled document using http://server/cgi-bin/djvuserve/dir/doc.djvu?bundled The keyword download inserts a content disposition HTTP header that suggests to display a save dialog instead of displaying the document. http://server/cgi-bin/djvuserve/dir/doc.djvu?download USING DJVUSERVE AS A HANDLER
The Apache web server provides a way to automatically execute djvuserve for all DjVu documents. This can be achieved using the following directives in either the Apache configuration file or the .htaccess files. Action djvu-server /cgi-bin/djvuserve/ AddHandler djvu-server .djvu Apache then executes program djvuserve for serving all DjVu files. Providing the URL of DjVu file serves this DjVu file as usual, except that bundled multipage documents are converted to indirect documents on the fly. This convenience comes at the expense of the computa- tional cost of executing djvuserve whenever a DjVu file is requested. TECHNICAL DETAILS
Program djvuserve provides a mean to directly access any component of a bundled multi-page DjVu document can be accessed using an extended URL. Suppose that the component file representing page 1 is named p0001.djvu. The following URL provides a direct access to this page: http://server/cgi-bin/djvuserve/dir/doc.djvu/p0001.djvu It is preferred however to access individual pages using the CGI style arguments described in nsdejavu(1), as in the following URL. http://server/cgi-bin/djvuserve/dir/doc.djvu?djvuopts&page=12 The special component file name index.djvu is recognized as a request for the index of the corresponding indirect multi-page document. In fact, when you access a bundled document using djvuserve, the browser gets redirected to the following URL: http://server/cgi-bin/djvuserve/dir/doc.djvu/index.djvu and then behaves as if the bundled file was a directory containing the various component files of an equivalent indirect document. ACCESS CONTROL
Program djvuserve, like many CGI programs, bypasses a number of access protections established in a web server. Assume for instance that your web site contains DjVu files protected by a password. Program djvuserve knows nothing about this protection and will happily serve any DjVu file associated with a valid URL. Access control with djvuserve can be implemented by first remembering that the web server always executes program djvuserve via shell script /var/www/cgi-bin/djvuserve. This script can decide to execute the real program djvuserve on the basis of the target filename available in the environment variable PATH_TRANSLATED. There can be several such scripts providing access to various collections of DjVu files. Each of these scripts can be password protected using the usual methods supported by your web server. KNOWN BUGS
Hyperlinks specified using a relative URL may not work with djvuserve. These URLs are relative to the URL of the DjVu document. Yet djvuserve changes the apparent document URL http://server/dir/doc.djvu into the more complicated URL http://server/cgi-bin/djvuserve/dir/doc.djvu/index.djvu. The extra components change the interpretation of relative URLs. CREDITS
This program was written by Leon Bottou <leonb@users.sourceforge.com>. SEE ALSO
djvu(1), djvmcvt(1), nsdejavu(1) DjVuLibre-3.5 01/22/2002 DJVUSERVE(1)
All times are GMT -4. The time now is 09:35 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy