sftp + expect: disconnection & restart removes already transfered data.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting sftp + expect: disconnection & restart removes already transfered data.
# 1  
Old 11-16-2010
sftp + expect: disconnection & restart removes already transfered data.

I have an ftp statement that when it restarts, it will write over the top of the file at the other end, rather than append to the file part sitting at the destination.

This is a problem because the flaky connection fails so regularly that the 2GB file I try to transfer will never complete.

Do any of you guru's out there understand how to ensure sftp will to recognise the file part at the other end and append to it?

And while we're at it, how can I improve reliability? When I run in batch mode it dies after less then 2MB sent, however if I run from a logged in shell and run sftp off the commandline, it will survive almost 1GB before exits. Why might this be?

Here's the code so far (naturally I've got a ~/.ssh/config file that has the proxy settings of the other end, and FYI I use the "corkscrew" library for this)

Code:
#!/bin/ksh
set timeout 60

result=1
tries=0
# TODO while [[ $result -ne 0 ]] ; do
while [[ 1 -ne 0 ]] ; do
   expect << EOF
      spawn sftp me@host.name.com <<EOF
      expect "\*password:\*"
      send -- "${password}\r"
      expect "sftp> "
      send -- "lcd /path/to/local/file/\r"
      expect "sftp> "
      send -- "cd /remote/path/to/place/file/\r"
      expect "sftp> "
      send -- "put file_name.txt\r"
      expect "sftp> "
      send -- "bye\r"
EOF
   result=$?
   let tries+=1 ;
   echo "###result:${result}:tries bottom:${tries}"
done

# 2  
Old 11-16-2010
You don't need to use expect, unless you just want to...
Code:
#!/bin/ksh

set -A A_FILE file1 file2 file3 file4 file5 file6 file7 file8 file9 file10

i=0

while [ -n "${A_FILE[$i]}" ]
do
   sftp -o IdentityFile=${KEYFILE} ${FTPUSER}@${FTPSERVER} <<-EOF
      put ${L_PATH}/${A_FILE[$i]} ${R_PATH}/${A_FILE[$i]}
      quit
   #
   # The EOF must be TABbed over.  It can not be spaces.
   # 
   EOF
   (( i = i + 1 ))
done

As for resuming a broken sftp session. I don't think sftp supports it,
but putty has pfstp and I think it does.
# 3  
Old 11-16-2010
Unfortunately we can't use identity files because we don't have access to a login shell or a home directory. "expect" was the only way.

I did some further reading for high reliability file transfer and discovered the crafty Russians have designed THE swiss army knife of file transfer "lftp"

I'll install and try this and report back.Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

SFTP Shell Script Get & Delete && Upload & Delete

Hi All, Do you have any sample script, - auto get file from SFTP remote server and delete file in remove server after downloaded. - only download specify filename - auto upload file from local to SFTP remote server and delete local folder file after uploaded - only upload specify filename ... (3 Replies)
Discussion started by: weesiong
3 Replies

2. Shell Programming and Scripting

Unable to do SFTP using expect

Hi, I am trying to sftp using expect, but not getting through as it prompt for password is coming Following is the code #/usr/bin/expect > output.log sftp medcdr@10.130.254.50 expect "password:" send "Med@Cdr12\n" expect "sftp>" send "put ZTE_*201505*\r" expect "sftp>" send "bye\r"... (7 Replies)
Discussion started by: siramitsharma
7 Replies

3. Shell Programming and Scripting

Problems with expect and sftp in bash

I'm having trouble with some automated sftp pulls. I'm using expect inside bash scripts and spawning SFTP. Some times the expect seems bog down. I have tried to put sleeps in my code to give everything time to work before I move on to next step but I till continue to get issues. For example when... (2 Replies)
Discussion started by: gosteen
2 Replies

4. Shell Programming and Scripting

Expect Script for SFTP Upload

I am attempting to utilize an expect script (that is called from a parent bash script) to perform a file transfer over sftp. The script works except I cannot catch timeouts. I need to be able to tell in the parent bash script when the expect script has timed out, or completed successfully. It... (3 Replies)
Discussion started by: thaller
3 Replies

5. Shell Programming and Scripting

Using expect to automate sftp

I am trying to use a for loop in my expect cmdFile that I am calling. I want to be able to call either one file name or a series of file names in the working directory (that I won't know the names before hand) and then pass the names to the sftp program. Something like for i in (ls *txt) do (0 Replies)
Discussion started by: vedder191
0 Replies

6. Shell Programming and Scripting

How to automate sftp without using expect script?

How to automate sftp with out using expect script? My batch file has the password but it is not taking. Please see below. I want to use this sftp connection in a loop for pushing new files in a directory one at a time. Hence I can not use an expect script. bash-2.05$... (5 Replies)
Discussion started by: Tuxidow
5 Replies

7. Shell Programming and Scripting

Help with Expect SFTP script

Hi All, Here is my Expect script, I don't get any error message when I run it. But the file never goes to other system? I also paste the output screen below. When I run the script, the script runs so fast. But when I do it manually, it takes about 10 minutes for the file to transfer. ... (1 Reply)
Discussion started by: samnyc
1 Replies

8. Shell Programming and Scripting

expect + cronjob + SFTP

Hi all, i got a really strange problem i wrote a script, when i run this script manually everything works fine but when i make a cronjob for it, with the same user, the EXPECT script will not work. Only the first line will be executed this is the SHell bash script #!/bin/sh; php -q... (2 Replies)
Discussion started by: digitac
2 Replies

9. Shell Programming and Scripting

Automating SFTP with Expect

Hello all, I've written an automated SFTP script to work with the Expect command. It recently occurred to me however, that if the client side box does not have the known host entry for the server, it will not work correctly. So I have added an expect for the known host prompt, and that part... (2 Replies)
Discussion started by: sysera
2 Replies
Login or Register to Ask a Question