Submit using curl


 
Thread Tools Search this Thread
Operating Systems Linux Ubuntu Submit using curl
# 1  
Old 06-02-2010
Submit using curl

I'm trying to upload a file to a page using curl and after uploading that file i want to store the redirecting page so i can download the results.
I'm using the command:

Code:
curl "http://apps.gdgps.net/kag_upload.php?kag_type=static&kag_frequency=dual&kag_latency=accurate&
kag_l1=P&kag_ema=7.5&rinex_interval=300&upfile_0=@CASC2270.09o&upload_button=Upload"

can anybody tell me what is wrong with this command?

Last edited by Scott; 06-03-2010 at 01:45 PM.. Reason: Please use code tags
# 2  
Old 06-03-2010
debug

The best way to debug this is to Wireshark the web browser packets. It would be easy for me to do this if you would post a simple input file to submit. JavaScript is used to submit the form and Wireshark will show what the actual page submitted is, whether any cookies are used, and whether there are any other hidden variables.

Once that is determined we can figure out what the curl command needs to be and then do a full trace with '--trace <file>'.

JavaScript can really complicate automation and sometimes completely prevent it.
# 3  
Old 06-03-2010
The input of the file is something like this:
Code:
     2.11           OBSERVATION DATA    M (MIXED)           RINEX VERSION / TYPE
teqc  2009Oct19                         20100212 17:06:33UTCPGM / RUN BY / DATE
EDVD                                                        MARKER NAME
EDVD                                                        MARKER NUMBER
Ricardo Pinho       ENTRE DOURO E VOUGA DIGITAL             OBSERVER / AGENCY
                                                            REC # / TYPE / VERS
30473663            TRM55971.00                             ANT # / TYPE
  4774346.5322  -712193.6930  4155295.3307                  APPROX POSITION XYZ
        0.0000        0.0000        0.0000                  ANTENNA: DELTA H/E/N
     1     1                                                WAVELENGTH FACT L1/2
     5    C1    L1    P2    L2    C2                        # / TYPES OF OBSERV
    30.0000                                                 INTERVAL
Forced Modulo Decimation to 30 seconds                      COMMENT
  2009     8    15     0     0    0.0000000     GPS         TIME OF FIRST OBS
    15                                                      LEAP SECONDS
Linux 2.4.20-8|Pentium IV|gcc -static|Linux|486/DX+         COMMENT
Spider V3,2,1,3225  EDVD                2009 08 15 01:00    COMMENT
THIS RINEX FILE IS CREATED FROM RTCM V3.0 DATA              COMMENT
Site Information :                                          COMMENT
São João da Madeira                                         COMMENT
Aveiro                                                      COMMENT
Portugal                                                    COMMENT
Eurasian                                                    COMMENT
Edifício Villa Balbina, Parrinho                            COMMENT
                                                            END OF HEADER
 09  8 15  0  0  0.0000000  0 12G11G14G17G19G20G23G28G32R03R18R19R20
  20204061.046   106173105.736 9  20204055.026    82732302.065 6
  24317182.598   127787671.463 6  24317177.518    99574813.950 1
  23267139.626   122269661.145 8  23267134.966    95275064.518 3
  22988089.228   120803264.966 8  22988082.808    94132424.697 4
  20444473.224   107436454.811 9  20444466.824    83716723.899 6
  23991862.240   126078089.385 8  23991855.620    98242667.813 1
  22921991.588   120455895.437 8  22921985.708    93861740.967 3
  20345088.586   106914198.779 9  20345083.826    83309773.986 6
  23099342.888   123652633.182 7  23099340.348    96174280.414 5
  22347917.172   119294734.537 8  22347917.112    92784802.964 6
  19275347.132   103110143.247 9  19275344.132    80196786.689 8
  20776460.724   111101100.392 8  20776459.684    86411972.037 7
 09  8 15  0  0 30.0000000  0 12G11G14G17G19G20G23G28G32R03R18R19R20
  20206296.646   106184854.498 9  20206290.986    82741456.941 6
  24329858.198   127854278.930 6  24329852.418    99626715.864 1


Last edited by Scott; 06-03-2010 at 01:44 PM.. Reason: Code tags, please...
# 4  
Old 06-03-2010
That's nice, but we still don't know what the actual page submitted is, whether any cookies are used, and whether there are any other hidden variables.
# 5  
Old 06-03-2010
The page is: Automatic Precise Positioning Service - APPS
I downloaded the page using curl and then run the program named formfind and the result was:
Code:
--- FORM report. Uses POST to URL "#"
--- type: multipart form upload [use -F]
Input: NAME="kag_type" VALUE="static" (RADIO)
Input: NAME="kag_type" VALUE="kinematic" (RADIO)
Input: NAME="kag_frequency" VALUE="single" (RADIO)
Input: NAME="kag_frequency" VALUE="dual" (RADIO)
Input: NAME="kag_latency" VALUE="nrt" (RADIO)
Input: NAME="kag_latency" VALUE="accurate" (RADIO)
Input: NAME="kag_l1" VALUE="CA" (RADIO)
Input: NAME="kag_l1" VALUE="P" (RADIO)
Input: NAME="kag_tdp" (CHECKBOX)
Input: NAME="kag_ema" VALUE="7.5" (TEXT)
Input: NAME="rinex_interval" VALUE="300" (TEXT)
Input: NAME="upfile_0" (FILE)
Input: NAME="reset_button" VALUE="Reset" (BUTTON)
Input: NAME="upload_button" VALUE="Upload" (BUTTON)
--- end of FORM

Moderator's Comments:
Mod Comment Please use code tags. Thank you

Last edited by Scott; 06-03-2010 at 01:47 PM.. Reason: Code tags
# 6  
Old 06-03-2010
cookies and javascript

Ran Wireshark (WS) and found out a couple things.
  1. You must be registered.
  2. It does use cookies.
  3. JavaScript generates a random ID.
  4. Upload file is multipart MIME format.
  5. Intermediate results for upload progress send JavaScript.
  6. Upload failed as I'm not registered and possibly the input file is incomplete.
I've attached the WS output as plain text. Sorry but it doesn't include the initial page request, just the submission of defaults with the input file you provided. Also set WS to only display HTTP messages. See attached file 'test15-ws-out.txt'

Using curl to accomplish this function appears to be rather complex. You'll have to simulate the JavaScript random ID to execute the "GET /kag_link_upload.php?rnd_id=1275597861022". You may be able to skip the upload progress "GET /kag_set_progress.php?upload_id=151cf1b0cd09e4cbd4fea32148619618" but you'll have to do MIME encoding on the "POST /cgi-bin/kag_upload.pl?upload_id=151cf1b0cd09e4cbd4fea32148619618" for the file. In each case you must set the proper "http.referer", probably provide a "http.user_agent" value which the site likes, and include the proper cookies.

A couple HTTP packets later my upload fails, so the rest of the trace isn't that relavent except in how to handle errors.

This task to fake this web browser app seems non-trivial to me. On How To Use APPS page it describes a way to upload the file to their FTP server where it will be automatically processed. An FTP transaction would be much easier to implement but you'll have to email them for login details. The other solution is to automate an email and FTP upload which seems to be a bit of a pain compaired to just FTPing a file.
This User Gave Thanks to m1xram For This Post:
# 7  
Old 06-03-2010
Thanks for your attention, this information was very helpful
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Curl to hit the submit button

Hello, I am looking to hit a URL using curl and click on submit button so that I can get the results. The below is the code <input name="tos_accepted" id="tos_accepted" class="button" type="submit" value="Yes, I Agree"/> <input name="tos_discarded" id="tos_discarded"... (1 Reply)
Discussion started by: Kochappa
1 Replies

2. Shell Programming and Scripting

Can't submit a form.

hello my script is submitting POST-data to a site (its not my first script, i've done these before many times (include parsing scripts) but this one is tough) so the problem is i'm submitting a form with firefox and in firebug i see WHAT exactly i'm submitting then when i do EXACTLY the... (28 Replies)
Discussion started by: tip78
28 Replies

3. Shell Programming and Scripting

Using cURL to submit a post form

I am trying to write a shell script to use curl in order to automate downloading data from a website. The URL with the post form is here: http://try-db.org/de/InfoBySpecies.php . I have a list of about 1800 different species I want to check. For Example, choose the first species and use the... (2 Replies)
Discussion started by: hansvg
2 Replies

4. UNIX for Dummies Questions & Answers

how to submit several commands together

Dear all, I have a simple question. I have many cut commands like below to run. I am wondering how to just type and submit all the commands at once and then each of the command will be run automaticaly one by one. Thanks a lot! cut -d ' ' -f 3-2002 ... (4 Replies)
Discussion started by: forevertl
4 Replies

5. Post Here to Contact Site Administrators and Moderators

Where to submit my scripts

have a doubt. Where I can submit shell scripts done by me ? (5 Replies)
Discussion started by: linuxadmin
5 Replies

6. News, Links, Events and Announcements

where to submit my work?

Hello sir, I worked out some features using java,shell programming and modified the bash shell. We have come up with 13 new features. I want to know where should I submit my work so that it would be of some use to the open source professionals :confused: (1 Reply)
Discussion started by: nsharath
1 Replies

7. Forum Support Area for Unregistered Users & Account Problems

cant submit anything

hello admins, i dont use unix.com very often - but when i do have questions, i submit them at the site - but now I cannot submit anything in any forum on the site. The is the only forum i could add a new thread. its tells me that my account could be de-righted, or that im tring to submit... (1 Reply)
Discussion started by: congo
1 Replies

8. Shell Programming and Scripting

Cannot submit a background job

Hi all, I am currently facing a problem when i am submitting a script to run in the background to collect statistics round the clock on an AIX box. I don't have root authority nor can I set it in cron. So when i submit the job, it runs fine, but won't let me signoff. It prompts me that... (2 Replies)
Discussion started by: tansha
2 Replies

9. Shell Programming and Scripting

How to submit cron jobs?

How can I write a script to submit a perl script as a cron job but only have it execute once? After it has executed once, I would like it to automatically insert itself again into cron. I want to avoid the situation where I schedule a cron job to run once a day, but end up with multiple... (4 Replies)
Discussion started by: siegfried
4 Replies
Login or Register to Ask a Question