Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Sftp files between servers

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Sftp files between servers
# 8  
Old 04-25-2013
Thanks For Your Reply!!!
Its a bit tricky!!! It should dump files per database into its own directory as shown below as we have 7 databases and create seperate logfiles:
Code:
${DUMPPATH}/${lv_dbname}

Tried the below script:
Code:
# Execute to setup global variables
. ${APPS}/db_batch/common/.batchenv
# *****************************************************************************
#  Function - Display parameter usage and terminate the job
# *****************************************************************************
display_usage()
{
   echo "==============================================================================="
   echo "Script Name:  ${CMD}"
   echo "Description:  Performs sFTP put of exported dump files"
   echo ""
   echo "Usage:"
   echo "ftp_dump_to_dr.sh [ -d <database_name> ] [ -x <execute flag> ]"
   echo "-d <database_name> - individual database name to ftp "
   echo "-x <execute flag> - yes or no (optional, default=no)"
   echo ==================================================
   echo "Required parameters:"
   echo "   lv_dbname      - name of database dump to sFTP"
   echo "   lv_rfc            - change number [ format CHGXXXX ]"
   echo "==============================================================================="
   exit 1
}
# *****************************************************************************
# Initialization of local variables
# *****************************************************************************
# Initialize job name
lv_job_name="sftp_dump_files"
# Initialize local job variables for directory path
lv_job_path="${DBBATCH}/backup"
lv_common_path="${lv_job_path}/common"
lv_shell_path="${lv_job_path}/shell"
lv_sql_path="${lv_job_path}/sql"
lv_log_path="${lv_job_path}/log"
mkdir -p ${lv_common_path} ${lv_shell_path} ${lv_sql_path} ${lv_log_path}
# Initialize local job variables for log files
lv_job_log_file="${lv_log_path}/${lv_job_name}_${lv_dbname}.log"
#  Setup local step variables
lv_step_name=sftp_dump_files
lv_step_log_file=${lv_log_path}/${lv_step_name}.log
# Additional local variables
lv_retcode=0
lv_exitcode=0
#*******************************************************************************
# Function - Check dumps file existence
#*******************************************************************************
check_dump_exist()
{
   dump_num=`ls -l ${DUMPPATH}/${lv_dbname}/${lv_dbname}_*.dump|wc -l`
   if [ ${dump_num} = 0 ]
   then
     echo "FAILURE - dumps ${DUMPPATH}/${lv_dbname}/${lv_dbname}_*.dump does not exist, the job has been terminated." >> ${lv_step_log_file}
     exit 1
   fi
}
#*******************************************************************************
# Function - Check ftp connection
#*******************************************************************************
check_ftp_connection()
{
   lv_tmpfile=/tmp/${ASENAME}_$$.out
   echo > ${lv_tmpfile}
   sftp -b ${lv_tmpfile} sybase@${SERVERNAME} >> ${lv_step_log_file} 2>> ${lv_step_log_file}
   lv_retcode=$?
   if [ ${lv_retcode} != 0 ]
   then
     echo "FAILURE - connection to sybase@${SERVERNAME} is no open, the job has been terminated." >> ${lv_step_log_file}
     exit 1
   fi
   rm -f ${lv_tmpfile}
}
# setup parameter variables
lv_tempfile="/tmp/tempfile"
# *****************************************************************************
#  Parse input parameter
# *****************************************************************************
set -- `getopt d:x: $* 2> ${lv_tempfile}.${ASENAME}`
if [ ${?} -ne 0 ]
  then
    echo "`cat ${lv_tempfile}.${ASENAME}`"
    display_usage
  else
    for i in ${*}
    do
      case ${i} in
        -d) lv_dbname="${2}"; shift 2 ;;
        -x) lv_execute="${2}"; shift 2 ;;
        --) shift; break ;;
      esac
    done
fi
if [ -z "${lv_dbname}" ]
then
    echo "Must provide db_name"
    display_usage
fi
if [ -z "${lv_execute}" ] || [ "${lv_execute}" = "no" ]
then
  lv_execute="no"
else
  if [ "${lv_execute}" != "yes" ]
  then
    echo "execute flag inner = ${lv_execute}"
    echo "Execute Flag cannot be [ ${lv_execute} ]. Execute Flag can only be either <yes|no>"
    display_usage
  fi
fi
#*****************************************************************************
# Parse Input Options Force to Lowercase
#*****************************************************************************
lv_dbname=`echo ${lv_dbname} | tr [:upper:] [:lower:]`
lv_execute=`echo ${lv_execute} | tr [:upper:] [:lower:]`

# *****************************************************************************
# Main part of the step script (sFTP put from ${LOCAL_HOST} to ${REMOTE_HOST})
# *****************************************************************************
Code:
#  sFTP dump file if it exists
if [ -f {DUMPPATH}/${lv_dbname}/${lv_dbname}_*.dump ]
then
   echo "Local Host: " ${LOCAL_HOST} | tee -a ${lv_step_log_file}
   echo "Local Files:" | tee -a ${lv_step_log_file}
   ls -lrt ${lv_dbname}_*.dump | tee -a ${lv_step_log_file}
   echo "" | tee -a ${lv_step_log_file}
   echo "Remote Host: " ${REMOTE_HOST} | tee -a ${lv_step_log_file}
 
   #  sftp the script if it is for EXPORT
   sftp sybase@${REMOTE_HOST} << EOF >> ${lv_step_log_file} 2>&1
      lcd ${DUMPPATH}/${lv_dbname}
      cd ${DUMPPATH}/${lv_dbname}}
      mput -p ${DUMPATH}/${lv_dbname}/${lv_dbname}*.dump
      ls -lart
   quit
EOF
else
   echo "FAILURE : Dump file ${DUMPPATH}/${lv_dbname}_${lv_dbname}.dump does not exist." | tee -a ${lv_step_log_file}
   echo "*** `date` Failed while executing ${CMD} ***" | tee -a ${lv_step_log_file}
   exit 1
fi

or

---------- Post updated at 10:36 AM ---------- Previous update was at 10:32 AM ----------

Script shoud do:
Dump files will be already existing in server and it should be sftp to another server .

We already know the dump path for each database ie /dump/sybase15.5/${lv_dbname}/*.dump

Reply would be appreciated Robin!!!

---------- Post updated 04-25-13 at 09:21 AM ---------- Previous update was 04-24-13 at 10:36 AM ----------

Hi Robin,
Any reply to my last post...Nothing seems to be working my way....
Last edited by nmm_dba; 04-24-2013 at 11:49 AM..
# 9  
Old 04-25-2013
I think you are missing the point, so apologies if I have not explained it clearly.

You still have an sftp command with input redirection:-
Code:
   #  sftp the script if it is for EXPORT
   sftp sybase@${REMOTE_HOST} << EOF >> ${lv_step_log_file} 2>&1
      lcd ${DUMPPATH}/${lv_dbname}
      cd ${DUMPPATH}/${lv_dbname}}
      mput -p ${DUMPATH}/${lv_dbname}/${lv_dbname}*.dump
      ls -lart
   quit
EOF

This is not allowed for SFTP. You do seem to use the right format in your function check_ftp_connection but I don't see where you try to create any commands in this file for your live attempt.

Can you adjust your code to the following (updates in red):-
Code:
# Execute to setup global variables
. ${APPS}/db_batch/common/.batchenv
# *****************************************************************************
#  Function - Display parameter usage and terminate the job
# *****************************************************************************
display_usage()
{
   echo "==============================================================================="
   echo "Script Name:  ${CMD}"
   echo "Description:  Performs sFTP put of exported dump files"
   echo ""
   echo "Usage:"
   echo "ftp_dump_to_dr.sh [ -d <database_name> ] [ -x <execute flag> ]"
   echo "-d <database_name> - individual database name to ftp "
   echo "-x <execute flag> - yes or no (optional, default=no)"
   echo ==================================================
   echo "Required parameters:"
   echo "   lv_dbname      - name of database dump to sFTP"
   echo "   lv_rfc            - change number [ format CHGXXXX ]"
   echo "==============================================================================="
   exit 1
}
# *****************************************************************************
# Initialization of local variables
# *****************************************************************************
# Initialize job name
lv_job_name="sftp_dump_files"
# Initialize local job variables for directory path
lv_job_path="${DBBATCH}/backup"
lv_common_path="${lv_job_path}/common"
lv_shell_path="${lv_job_path}/shell"
lv_sql_path="${lv_job_path}/sql"
lv_log_path="${lv_job_path}/log"
mkdir -p ${lv_common_path} ${lv_shell_path} ${lv_sql_path} ${lv_log_path}
# Initialize local job variables for log files
lv_job_log_file="${lv_log_path}/${lv_job_name}_${lv_dbname}.log"
#  Setup local step variables
lv_step_name=sftp_dump_files
lv_step_log_file=${lv_log_path}/${lv_step_name}.log
# Additional local variables
lv_retcode=0
lv_exitcode=0
#*******************************************************************************
# Function - Check dumps file existence
#*******************************************************************************
check_dump_exist()
{
   dump_num=`ls -l ${DUMPPATH}/${lv_dbname}/${lv_dbname}_*.dump|wc -l`
   if [ ${dump_num} = 0 ]
   then
     echo "FAILURE - dumps ${DUMPPATH}/${lv_dbname}/${lv_dbname}_*.dump does not exist, the job has been terminated." >> ${lv_step_log_file}
     # exit 1
     return 1
   fi
}
#*******************************************************************************
# Function - Check ftp connection
#*******************************************************************************
check_ftp_connection()
{
   lv_tmpfile=/tmp/${ASENAME}_$$.out
   # echo > ${lv_tmpfile}
   > ${lv_tmpfile}
   sftp -b ${lv_tmpfile} sybase@${SERVERNAME} >> ${lv_step_log_file} 2>> ${lv_step_log_file}
   lv_retcode=$?
   if [ ${lv_retcode} != 0 ]
   then
     echo "FAILURE - connection to sybase@${SERVERNAME} is no open, the job has been terminated." >> ${lv_step_log_file}
     # exit 1
     return 1
   fi
   rm -f ${lv_tmpfile}
}
# setup parameter variables
lv_tempfile="/tmp/tempfile"
# *****************************************************************************
#  Parse input parameter
# *****************************************************************************
set -- `getopt d:x: $* 2> ${lv_tempfile}.${ASENAME}`
if [ ${?} -ne 0 ]
  then
    echo "`cat ${lv_tempfile}.${ASENAME}`"
    display_usage
  else
    for i in ${*}
    do
      case ${i} in
        -d) lv_dbname="${2}"; shift 2 ;;
        -x) lv_execute="${2}"; shift 2 ;;
        --) shift; break ;;
      esac
    done
fi
if [ -z "${lv_dbname}" ]
then
    echo "Must provide db_name"
    display_usage
fi
if [ -z "${lv_execute}" ] || [ "${lv_execute}" = "no" ]
then
  lv_execute="no"
else
  if [ "${lv_execute}" != "yes" ]
  then
    echo "execute flag inner = ${lv_execute}"
    echo "Execute Flag cannot be [ ${lv_execute} ]. Execute Flag can only be either <yes|no>"
    display_usage
  fi
fi
#*****************************************************************************
# Parse Input Options Force to Lowercase
#*****************************************************************************
# lv_dbname=`echo ${lv_dbname} | tr [:upper:] [:lower:]`
# lv_execute=`echo ${lv_execute} | tr [:upper:] [:lower:]`
typeset -u lv_dbname lv_execute


# *****************************************************************************
# Main part of the step script (sFTP put from ${LOCAL_HOST} to ${REMOTE_HOST})
# *****************************************************************************

#  sFTP dump file if it exists
if [ -f {DUMPPATH}/${lv_dbname}/${lv_dbname}_*.dump ]
then
   echo "Local Host: " ${LOCAL_HOST} | tee -a ${lv_step_log_file}
   echo "Local Files:" | tee -a ${lv_step_log_file}
   ls -lrt ${lv_dbname}_*.dump | tee -a ${lv_step_log_file}
   echo "" | tee -a ${lv_step_log_file}
   echo "Remote Host: " ${REMOTE_HOST} | tee -a ${lv_step_log_file}
 
   #  sftp the script if it is for EXPORT
   # sftp sybase@${REMOTE_HOST} << EOF >> ${lv_step_log_file} 2>&1
   #   lcd ${DUMPPATH}/${lv_dbname}
   #   cd ${DUMPPATH}/${lv_dbname}}
   #   mput -p ${DUMPATH}/${lv_dbname}/${lv_dbname}*.dump
   #   ls -lart
   #quit
#EOF

   lv_tmpfile=/tmp/${ASENAME}_$$.out
   echo "lcd ${DUMPPATH}/${lv_dbname}
         cd ${DUMPPATH}/${lv_dbname}
         mput -p ${DUMPATH}/${lv_dbname}/${lv_dbname}*.dump
         ls -lart
         quit" > ${lv_tmpfile}
   sftp -b ${lv_tmpfile} sybase@${REMOTE_HOST} 2>&1 >> ${lv_step_log_file}
   rm ${lv_tmpfile}
else
   echo "FAILURE : Dump file ${DUMPPATH}/${lv_dbname}_${lv_dbname}.dump does not exist." | tee -a ${lv_step_log_file}
   echo "*** `date` Failed while executing ${CMD} ***" | tee -a ${lv_step_log_file}
   exit 1
fi

.... and let us all know how you get on.

Any trace output would be useful.



Robin
Last edited by rbatte1; 04-25-2013 at 10:58 AM.. Reason: SFTP batch file cleanup added.
This User Gave Thanks to rbatte1 For This Post:
nmm_dba
# 10  
Old 04-25-2013
Thanks Robin,
Will try the script. However, i was wondering, why we require mput and then sftp as shown below because the dump will be already existing in the local server for 8 databases under each database directories:

Code:
echo "lcd ${DUMPPATH}/${lv_dbname}
         cd ${DUMPPATH}/${lv_dbname}
         mput -p ${DUMPATH}/${lv_dbname}/${lv_dbname}*.dump
         ls -lart
         quit" > ${lv_tmpfile}
   sftp -b ${lv_tmpfile} sybase@${SERVERNAME} 2>&1 >> ${lv_step_log_file}

Will this script take care for all the database. Because each db has its own directory under its main directory:
$DUMPPATH/dbname

eg:
$DUMPPATH/central
$DUMPPATH/ltts
etc..
# 11  
Old 04-25-2013
A plain put command to SFTP and or FTP will probably mean that the * will be interpreted as a literal character. As such it will probably not be found. Using mput does the file name expansion for you. It doesn't necessarily mean that you will transfer more than one file, but it also means that you have to send the file as the same name.

Something like:-
Code:
mput fileA* targetfile

will send all the files is finds named fileA* as the same names and then you will get an error when it tries to send targetfile, unless that file exists locally, when it will send that too.


Do some experimenting to see what effect is has with your installation combination of client and server to be sure. They should all match this standard though, I think, however there are variation defending on the default globbing value - see your man page for ftp for details.






Robin
Liverpool/Blackburn
UK
Last edited by rbatte1; 04-25-2013 at 12:51 PM.. Reason: Spllelnig mitsake
This User Gave Thanks to rbatte1 For This Post:
nmm_dba
# 12  
Old 04-25-2013
Ok Thanks.
Files are unique for all DBS except database name as shown elow:

[msstrvdbuxtr02]:[sybase]: /dump/sybase15.5/central
Code:
AIX:> ls
-rw-r-----    1 sybase   dbo       250138624 Apr 24 20:07 central_4of4.dump
-rw-r-----    1 sybase   dbo       248127488 Apr 24 20:07 central_3of4.dump
-rw-r-----    1 sybase   dbo       248363008 Apr 24 20:07 central_2of4.dump
-rw-r-----    1 sybase   dbo       246398976 Apr 24 20:07 central_1of4.dump

[sybase]: /dump/sybase15.5/central
[sybase]: /dump/sybase15.5/workdb
Code:
AIX:> ls
-rw-r-----    1 sybase   dbo        75139072 Apr 24 20:08 workdb_4of4.dump
-rw-r-----    1 sybase   dbo        78024704 Apr 24 20:08 workdb_3of4.dump
-rw-r-----    1 sybase   dbo        78407680 Apr 24 20:08 workdb_2of4.dump
-rw-r-----    1 sybase   dbo        77838336 Apr 24 20:08 workdb_1of4.dump

Last edited by Franklin52; 04-26-2013 at 03:30 AM.. Reason: Please use code tags
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Connect direct - SFTP - List of servers that I can connect

Greetings Experts, I am working for a bank client and have a question on connect-direct and SFTP. We are using Linux RedHat servers. We use connect-direct to transfer (NDM) files from one server to another server. At times, we manually transfer the files using SFTP from one server to another... (2 Replies)
Discussion started by: chill3chee
2 Replies

2. UNIX for Advanced & Expert Users

How to setup sftp beteen two servers?

Hi Could you please help me out how to configure between two server I don't have admin idea to setup the Sftp server the requirements is we want to send a file to vendor so we need sftp configuration so that can we can send file through sftp Please let me know what should I ask to vendor... (1 Reply)
Discussion started by: jagu
1 Replies

3. Shell Programming and Scripting

Check files and archive the files using sftp

Hi, I have to check the files in another server using sftp to do that, below is the code i am going with #!/bin/bash export SRC_FOLDER=$1 export ARC_FOLDER=$2 HOST=as07u3456 USER=relfag sftp ${USER}@${HOST} <<EOF cd $SRC_FOLDER/DSCOR ls bye EOF echo "done" whatever the files i... (8 Replies)
Discussion started by: ursrami
8 Replies

4. Shell Programming and Scripting

Renaming multiple files in sftp server in a get files script

Hi, In sftp script to get files, I have to rename all the files which I am picking. Rename command does not work here. Is there any way to do this? I am using #!/bin/ksh For eg: sftp user@host <<EOF cd /path get *.txt rename *.txt *.txt.done ... (7 Replies)
Discussion started by: jhilmil
7 Replies

5. Red Hat

Chroot sftp users, remote sftp login shows wrong timestamp on files

Hello, I have a weird issue, I have RHEL 5.7 running with openssh5.2 where sftpgroup OS group is chroot. I see the difference difference in timestamp on files, when I login via ssh and SFTP, I see four hour difference, is something missing in my configuration. #pwd... (8 Replies)
Discussion started by: bobby320
8 Replies

6. Shell Programming and Scripting

SFTP files to two servers

Hi, I need to write a unix shell script which sends files to a server1 from my server via sftp. Then it needs to again SFTP files from that server1 to another server2. And finally remove the transferred files in the intermediate server1. Please provide me pointers on how to achieve the same. All... (3 Replies)
Discussion started by: vharsha
3 Replies

7. UNIX for Dummies Questions & Answers

Need to calculate sftp time beteen two servers.

Hi, I need to calculate time taken by a 100kb file for sftp between two servers. I am new to unix can some body please tell me how do this. Thanks in advance... Vivek (2 Replies)
Discussion started by: koulvivek
2 Replies

8. UNIX for Dummies Questions & Answers

Unable to scp/sftp between two servers

I have four servers that for all intents and purposes are the same (I have the same profile on all four), North, South, Brooklyn & Queens. I have a script that scp's a file from Queens to brooklyn, and it runs just fine. I tried to replicate the script on South, to transfer a file to North, and... (1 Reply)
Discussion started by: DeCoTwc
1 Replies

9. Shell Programming and Scripting

SFTP Transfer Across Multiple Servers

Hello everyone, Have a question which I'm not entirely sure about, but will ask anyway. We have three servers (A/B/C), 'A' being the local host, 'B' being a SFTP server on the DMZ, and 'C' being the intended remote host. The task is to transfer file/s from server 'A' to server 'C' via server... (0 Replies)
Discussion started by: Cameron
0 Replies

10. Shell Programming and Scripting

need script to connect sftp servers

Dear friends, i need to connect sftp server from my home directory using script . Please can anyone help me on this. (1 Reply)
Discussion started by: kittusri9
1 Replies
Login or Register to Ask a Question