FTP/SFTP/SCP Files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting FTP/SFTP/SCP Files
# 1  
Old 02-01-2012
FTP/SFTP/SCP Files

Hi,

I have a process which FTP's the files from one server to another server.
Sometimes only half or a part of the file is delivered to remote location, but on the end log says FTP is successful. But ideally file in full is not delivered to remote location.
How can i catch these kind of errors in FTP as well as in SFTP?
How can i make sure file is delivered in full to remote location.
Any Checksum or calculating file size of remote location would help?
# 2  
Old 02-01-2012
For file transfers "ftp" or "sftp" is usually a last resort when transferring files to incompatible or isolated platforms. If you have the opportunity to use "rcp" or "scp" backed with processing in Remote Shell this is preferable.

Assuming you only have "ftp" or "scp" the common techniques are:
1) Transfer each file under a temporary name and then rename the files to their correct names after the last transfer has finished.
2) Include a verifiable control record in the data file design.
3) Transfer a batch header file first which itemises filenames and checksums for the remote computer to check. (This assumes some compatibility between the computers).
4) Compress the file before transfer into a verfiable archive.

Obviously your "ftp" script needs proper error handling but I'd first look at why the transfers are failing and then how to deal with it. In my experience the most common "ftp" problem is the "mysterious hang" or "crawling speed" when transferring across a modern high-speed WAN.

Ps. Just scanned some of your earlier posts. Can you post the actual script because I see no error handling?
# 3  
Old 02-02-2012
Thanks for the response. Please see my FTP code snippet below
Code:
for trial in {1..5} # Retry FTP 5 times
   do
     echo -e "\nAttempt to FTP the file ==> $trial" >> $upload_log;
     /usr/bin/ftp -niv $ftp_host > ${work_dir}/${abc}_ftp_temp.txt 2>&1 <<EndFtp
      quote user $ftp_user
      quote pass $ftp_pass
      cd $ftp_dir
      bin
      put $upload_file_name
EndFtp
      cat ${work_dir}/${abc}_ftp_temp.txt >> $upload_log;
      egrep -ri -h "^226" ${work_dir}/${abc}_ftp_temp.txt > /dev/null; 
# Transfer Check
      found=`echo $?`;
      if [ $found -eq 0 ]; then 
 # If FTP is successfull, break the loop and try FTP for next file
         break;
         elif [ $trial -lt 5 ]; then
           sleep 60;  # Halt the process for a minute
      fi
  done

FTP log is written to a temp file. After FTP is complete, code will search for string starting with 226 ( FTP Code for transfer success ) and if not available, then do a FTP retry. If 226 is available, loop is exited.

Here the problem is 226 code is possible for half transferred files too..So i am trying to find a way to find out part of the file transferes and re-transfer them.

Last edited by Franklin52; 02-03-2012 at 01:57 PM.. Reason: Please use code tags for data and code samples, thank you
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How can we overcome Broken pipe error during scp,SFTP,Rsync while transferring big files.?

Hello All, Hope all are doing well. We use scp (some times sftp and rsync also) for transferring big files (around 2GB each ) from 1 Network to another Network. The Issues which we face :- During transfer some times( Once in 1 week (or twice)) , the speed of transfer gets down to 30 kb/s,... (2 Replies)
Discussion started by: Upendra Bhushan
2 Replies

2. IP Networking

How to transfer files from UNIX server to windows machine or vice versa using ftp or sftp commands?

hi, i want to write a shell script code which transfers files from a directory in unix server to a directory in a windows machine.. can any1 give me a sample code which uses ftp or sftp command.. thanks very much, (3 Replies)
Discussion started by: Little
3 Replies

3. Solaris

How do files transferred via ftp or sftp gets thier permissions at destination?

We have umask defined under /etc/.login as 022. I have my user specific umask defined in /userhome/.login as 002. I understand ftp will not execute anything at destination, it simply transfers files. But it seem to be using 022 as umask for the files transferred. How does ftp knows what umask... (4 Replies)
Discussion started by: kchinnam
4 Replies

4. Shell Programming and Scripting

Transfer files from linux server to windows using secure ftp (sftp)

HI, I have to transfer files from linux server to windows using secure ftp (sftp) .Kindly help me out. (3 Replies)
Discussion started by: manushi88
3 Replies

5. Shell Programming and Scripting

SFTP and SCP help

Hi All, I have to make an automated script that needs to do SFTP or SCP from my Unix server to another Unix server. I have gone through search of this website for sftp and scp. I really get confused when it talk about ssh key or rsh key and sftp -b thing I would really appreciate if... (4 Replies)
Discussion started by: pinnacle
4 Replies

6. SuSE

scp/sftp only files which are not found.

Hi, I need to write a script to scp/sftp files from server A to server B..But i do not want to overwrite the files which already exist on server B. Anyone has a script which does the above Regards (2 Replies)
Discussion started by: SystemEng
2 Replies

7. Shell Programming and Scripting

Error copying files from Unix (Solaris10) to Windows Server 2003 using scp/sftp

Hi, I have generated a Public/Private Key Pair in Solaris Unix (source) server and deployed the Public key in Windows 2003(target) server .ssh directory of user profile. When i try to connect(ssh, scp, sftp) from Unix, i'm getting below error message. Sun_SSH_1.1, SSH protocols 1.5/2.0,... (0 Replies)
Discussion started by: ajaykumarb
0 Replies

8. AIX

what to use sftp or scp

hi, i have a weird problem i have to copy the file with caret(^) in it. but when i tries to copy with scp. It(scp) says that it cant use ^file_name scp mohit^narang user@machine/mohit^narang the error comes in the second parameter.if i used user@machine/mohit_narang(under score) instead... (2 Replies)
Discussion started by: narang.mohit
2 Replies

9. UNIX for Dummies Questions & Answers

File Missing When Grabbing Files from SFTP Server using SCP Command

Hi, I have this problem where sometimes my files would go missing when I schedule my crontab to run the SCP command to get file from the SFTP server. My crontab will run the scripts at an interval of 3 minutes (between the two scripts) The following is the setting in my crontab. ... (1 Reply)
Discussion started by: gingervitus
1 Replies

10. UNIX for Advanced & Expert Users

sftp vs scp

My transmit rates are waaay faster using scp over sftp....anyone know why scp is faster than sftp? I am using solaris 8 for my unix systems. -S (2 Replies)
Discussion started by: Sowser
2 Replies
Login or Register to Ask a Question