Shell Programming and Scripting

Hi All,

I have a pretty demanding requirement for an SFTP script I have been trying to put together.

I have nearly 100 files (all with the names staring with T_PROD) generated in my local server daily. I need to transfer each of these files to a remote server via SFTP (that's a client demand, can't use SSH or others). Since these files are pretty big (approx 10 MB each), we need to have a check on succesful transfer of each of the files. In case a file is not transferred properly, we will be trying to re-send the file, at max 3 times.

Since it's SFTP, we're not allowed to execute shell commands in the remote server. I tried creating the md5sum hash values, but stuck thereafter. How do we identify the exact files those failed the sanity check, and try re-sending them?

A barebone structure of the script is as below

Code:

#!/bin/bash
. ./params # parameter file

cd output

sftp ${FTPUSER}@${FTPSERVER} << EOF

cd $FTPFOLDER
put T_PROD*
!md5sum * > chklist.md5

bye
EOF

I'm running this script in Linux kernel 2.6.39

Do let me know if any further information is required.

- Best,

Avik

Welcome Aviktheory11,

Your sftp session is not a normal shell session for running commands remotely so it is a little awkward. You command below

Code:

!md5sum * > chklist.md5

will run on the local server, so I'm not sure that is what you really want.

If you were using plain ftp then you would get a zero return code if the connection opened and you had to resort to switching on flags to get more verbose output that can be captured in a log file and read afterwards. sftp on the other hand is better with errors. Could you do something like the following instead?:-

Code:

#!/bin/bash
. ./params # parameter file

cd output

errcnt=0
for file in T_PROD*
do
   sftp ${FTPUSER}@${FTPSERVER} << EOF
      cd $FTPFOLDER
      put $file
        EOF
# Note not an indentation with spaces on previous line.  Used tab for clarity.
   if [ $? -ne 0 ]
   then
      printf "Something went wrong with file \"$file\"\n"
      ((errcnt=$errcnt+1))
   else
      printf "File \"$file\" send successfully.\n"
   fi
done

if [ $errcnt -ne 0 ]
then
   printf "ERROR in $errcnt transfers!\n"
done

I'd say sftp is a secure and safe protocol. It either succeeds or it doesn't. Run it with raised logging level (-v option) . Check the exit code after completion.

Why not just use scp?

Run it for each file and check the exit code, log errors as appropriate.

Hi achenle, as I've said in my first thread, using SFTP is a business requirement, and we can't help abiding by that.





Thanks for the help rbatte1. However, will just checking the return code of sftp ensure the correct transfer of files ? I'm dealing with file transfer things just recently [that's why I've used the md5sum, din't realise it worked only locally], and I believe the return code is 0 even if a file was transferred partially. Please correct me if I'm wrong.

I'll try out your suggestion though. Thanks a lot for the help again !!!

- Best,

Avik

You could also list the file catching the output and comparing it later. Use an embedded dir $file in your script after you send the file. You can compare the number of bytes with the original file.

Unfortunately in this case sftp is pretty robust. I will see if I can test it out by sending a large file and then killing the sftpd process on the remote side to see what happens when I get a chance (unless anyone else would care to have a go first)



Robin

If you want to run commands remotely, ssh is your best bet. sftp is not a shell. Neither is scp.

Look -- sftp is scp. It's the same protocol. If a server supports sftp it supports scp and vice versa.