Shell Programming and Scripting

Hey everyone. First, let me preface this post by stating that there are some bad things I'm doing. I'm using python to work around SCP's refusal to take standard input for a password. The reason for doing this as opposed to using keys is because the servers are on lock down due to them being in our test lab, and they're not to be touched it it doesn't mirror production. So because of the amount of beaurocracy that I would have to go through, even though I already have access to the data I need, I just need a workaround - a way to copy the data to a processing server that doesn't involve me being the man in the middle.

Here is the script as it stands:

Code:

#!/usr/bin/python
import pexpect

PASS='password'
COMMAND="scp -oPubKeyAuthentication=no admin@10.1.2.3:/var/path/to/files/*.csv /home/local/path/to/files/"

child=pexpect.spawn(COMMAND)
child.expect('Password:')
child.sendline(PASS)
child.expect(pexpect.EOF)
print child.before

Now, this currently grabs every CSV file in the directory on the remote server, which is fine for grabbing lab data. However, if a similar solution needs to be used for production, we'll be dealing with significantly larger files... each one upwards of 100MB, so the overhead traffic could become immense.

What I would like to do is find a way to ignore files that have already been downloaded, and only download new/updated files. Is this possible? Any help would be appreciated.

You can fetch the list of *.csv files on server the same way you're doing: instead of scp, invoke this

Code:

FETCHCOMMAND="echo 'ls /var/path/to/files/*.csv' | ssh admin@10.1.2.3"

Then capture the output and check for existence of each file on the local machine.
You may also consider the -C option (compression) of scp (ssh).
The single quotes are essential to prevent shell globbing.