I have a client that needs some data downloaded from a website periodically. The problem is that the data is not available directly (i.e. ftp) and must be accessed via logging in to the site, selecting the data manually, then finally clicking a link to grab the zip file. The site does not have an API to make this an easy task, it must all be done via clicking links.
I've been tasked with automating this process, however I'm unsure which scripting language would be a good fit. I'm not familiar with perl or python, though from what I've been reading I'm leaning towards python. I doubt this is something I can accomplish easily via bash scripting (where I'm most comfortable), but then I've seen some really clever stuff done with bash scripts so it's not out of the question.
you know this can proobably be done with a few lines of bash and wget.
when you log in, you send a POST command with some variables. It is possible to see the string that is being sent using the firefox plugin livehttpheader. You can then copy-paste the datastring being sent and use
This is how I login to unix.com (i copy-pasted the post-data string and url from livehttpheaders):
the scr.html now contains a redirect page, and you can just extract the url on the redirect page:
and visit the logged in unix.com:
use the same principles on the login page. Sometimes there are "hidden" variables being sent, but they can be extrcted from the source using the same principle as I used to extract the new url above (save page source into an html file and extract using awk).
Only in a few unusual cases you have to simulate a browser and code every textbox and button being clicked. And then you can use something like perl mechanize as has been mentioned.
There is a python script that I would like converted to a perl script. If someone has the time to convert the script I would appreciate it. You can find the script below:
reboot-mb8600/reboot-mb8600.py at master . j4m3z0r/reboot-mb8600 . GitHub
#!/usr/bin/python
'''
A hacky script to... (1 Reply)
Hi All,
I'm confused what to use (Python/Perl/Shell), I have a scenario here wherein i need to process a data which might have millions of records in tabular format, my task is to find the {0,null,NA} in each and every column and also inform the end user that this column has this many values... (2 Replies)
Anyone care to give me the cliff notes as to why one would want to use perl vs python? Unix Solaris over here. Have capability to run both perl and python and bash/ksh and others.
Now what I am interested in using unix and its programs for large amounts of data, sql (done in Unix SAS)...etc.... (5 Replies)
How to convert below bash code in perl and python.
for BLOCK in /sys/block/emcpow*
do
echo "100000" > "$BLOCK"/queue/nr_requests
echo "noop" > "$BLOCK"/queue/scheduler
done (2 Replies)
Hello all,
I am in a bit of dilema here. I want to implement a suite. it involves a lot of text file processing and shell scripts . which one to use perl or python ? (4 Replies)
I have the following problem, which I need done in Perl/ or Python using Unix/linux filters...
1. You have a very large file, named 'ColCheckMe', tab-delmited, that you are asked to process. You are told that each line in 'ColCheckMe' has 7 columns, and that the values in the... (1 Reply)
I have an task definition listing xml file that contains a list of tasks such as
<TASKLIST
<TASK definition="Completion date" id="Taskname1" Some other
<CODE name="Code12"
<Parameter pname="Dog" input="5.6" units="feet" etc /Parameter>
<Parameter... (3 Replies)
I have a file contains
TASK gsnmpproxy {
CommandLine = $SMCHOME/bin/gsnmpProxy.exe
}
TASK gsnmpdbgui {
CommandLine = $SMCHOME/bin/gsnmpdbgui.exe
I would like to comment and than uncomment specific task eg TASK gsnmpproxy
Pls suggest how to do in shell script (9 Replies)