Shell Programming and Scripting

I have a client that needs some data downloaded from a website periodically. The problem is that the data is not available directly (i.e. ftp) and must be accessed via logging in to the site, selecting the data manually, then finally clicking a link to grab the zip file. The site does not have an API to make this an easy task, it must all be done via clicking links.

I've been tasked with automating this process, however I'm unsure which scripting language would be a good fit. I'm not familiar with perl or python, though from what I've been reading I'm leaning towards python. I doubt this is something I can accomplish easily via bash scripting (where I'm most comfortable), but then I've seen some really clever stuff done with bash scripts so it's not out of the question.

Any thoughts?

I don't know how to do it in Python, but I know that it can be fairly easily (for Perl programmer) done in Perl with WWW::Mechanize module.

you know this can proobably be done with a few lines of bash and wget.

when you log in, you send a POST command with some variables. It is possible to see the string that is being sent using the firefox plugin livehttpheader. You can then copy-paste the datastring being sent and use

Code:

wget --post-data "username=afagr&password=lkjafd" --save-cookies cookie.txt --keep-session-cookies cookie.txt $url

This is how I login to unix.com (i copy-pasted the post-data string and url from livehttpheaders):

Code:

wget --save-cookies cookie.txt --keep-session-cookies cookie.txt \
--post-data "vb_login_username=locoroco&vb_login_password=&s=&securitytoken=guest&do=login&\
vb_login_md5password=pae4&vb_login_md5password_utf=4gliuh" \
-q -O- https://www.unix.com/login.php?do=login > src.html

the scr.html now contains a redirect page, and you can just extract the url on the redirect page:

Code:

url=$(awk -v FS="\"" '/php\?s=/ && /window.location/{print $2}' s.html)

and visit the logged in unix.com:

Code:

wget --load-cookies cookie.txt -q -O- $url > loggedin.html

use the same principles on the login page. Sometimes there are "hidden" variables being sent, but they can be extrcted from the source using the same principle as I used to extract the new url above (save page source into an html file and extract using awk).

Only in a few unusual cases you have to simulate a browser and code every textbox and button being clicked. And then you can use something like perl mechanize as has been mentioned.