Sponsored Content
Top Forums Shell Programming and Scripting Archiving files using shell script Post 302947333 by Rads on Thursday 18th of June 2015 05:11:27 AM
Old 06-18-2015
Hi Robin,

Thanks for the information. I am new to shell scripting and this script for archiving is a bit challenging for me. Your help would be much appreciated. Please find the code as below:
Code:
#!/bin/ksh
# Purpose                      : Transfer files from Local to Remote/Remote to 
#Name of the file that is executed
SCRIPT_NAME=`basename ${0} .ksh`    
#Lock file/Temporary file to ensure that the files are not run multiple times                                         
LCK_FILE="/tmp/${SCRIPT_NAME}_${2}.run"  
#File used for build of SFTP tasks                                    
TMP_SFTP_SCRPT="/tmp/sftp_script_${2}_$$.ftp"  
#Denotes the current directory                              
DIR_NAME=`dirname ${0}`    
#File that defines the source and target specifications                                                  
FILE_CONFIG="${DIR_NAME}/${1}.config"                                        
FOUND=false
TIMESTAMP=`date '+%d-%b-%Y_%R'`
FILES_PATH="/tmp/Files2Transfer_${2}"

#
# Copy files from local system to remote system
#
Local2Remote() {

	cd ${FILES_PATH} 2>/dev/null
	if [[ ${?} -gt 0 ]];then
          print -u2 "LocalTarget path not found";
          rm ${LCK_FILE}
          exit -1;
        fi

	echo cd ${FILE_TARGET_PATH} > ${TMP_SFTP_SCRPT}
	echo mput ${FILE_MASK} >> ${TMP_SFTP_SCRPT}
	echo bye >> ${TMP_SFTP_SCRPT}

	sftp -oPort=${FILE_TARGET_PORT} -b ${TMP_SFTP_SCRPT} ${FILE_TARGET_ACCOUNT} 2>/dev/null 1>&2

	rm ${TMP_SFTP_SCRPT}

}

#
# Copy files from remote system to local system
#
Remote2Local() {

	cd ${FILES_PATH} 2>/dev/null
	if [[ ${?} -gt 0 ]];then
          print -u2 "LocalSource path not found";
          rm ${LCK_FILE}
          exit 1;
        fi

	echo cd ${FILE_SOURCE_PATH} > ${TMP_SFTP_SCRPT}
	echo mget ${FILE_MASK} >> ${TMP_SFTP_SCRPT}
	
# The below line needs to be modified to enable archive functionality. 	
  echo rm ${FILE_MASK} >> ${TMP_SFTP_SCRPT}    

# Added below line to copy and remove the files to the desired folder 
# echo mv ${file_mask} >> ${tmp_sftp_scrpt}                                  	
		          
# End of change 
	echo bye >> ${TMP_SFTP_SCRPT}

	sftp -oPort=${FILE_SOURCE_PORT} -b ${TMP_SFTP_SCRPT} ${FILE_SOURCE_ACCOUNT} 2>/dev/null 1>&2

	rm ${TMP_SFTP_SCRPT}
			
}

#
# Main function
#

if [ ! -f ${FILE_CONFIG} ]; then
	print -u2 "Configuration file not found"
	rm ${LCK_FILE}
        exit 1
fi

if [ -f ${LCK_FILE} ]; then
	print -u2 "Script with parameter ${CL_NAME} already running, exiting..."
	exit 1
else
	touch ${LCK_FILE}
fi

exec < ${FILE_CONFIG}
	while IFS=\| read -r name source_account source_port source_path target_account target_port target_path mask; do
	case ${name} in
		\#* | "")
			continue
			;;
	esac
		
			eval FILE_SOURCE_ACCOUNT=${source_account}
                        eval FILE_SOURCE_PORT=${source_port}
			eval FILE_SOURCE_PATH=${source_path}
			eval FILE_TARGET_ACCOUNT=${target_account}
                        eval FILE_TARGET_PORT=${target_port}
			eval FILE_TARGET_PATH=${target_path}
			eval FILE_MASK=${mask} 
			FOUND=true
			break

done

if [ ${FOUND} = false ]; then
	print -u2 "File to transfer not configured"
    rm ${LCK_FILE}
	exit 2
fi

mkdir ${FILES_PATH}

set -f
Remote2Local
Local2Remote
set +f

rm -Rf ${FILES_PATH}

rm ${LCK_FILE}

exit 0

I am following below process to test the files:
This script and the config file exists in one server. I am creating source and target folder in FTP server and putting in a sample CSV in source folder. I then login via putty and execute this script name.ksh<space> "config file name". Upon execution, I should be able to see the CSV file in the target folder.

I would like to know if I am using the correct method to do the archival process. Also is it possible to do the logging during the archival process (i.e., between archive start and end) to include name, timestamp information?

---------- Post updated 06-18-15 at 11:11 AM ---------- Previous update was 06-17-15 at 04:07 PM ----------

Hello,

I have now modified the script to include a functionality to archive the files. Please find the code as below. Could somebody suggest how I can log the information (start time of archival, files archived, folder to which files are moved, archival end time) in the script?
Code:
CL_NAME=${1}
SCRIPT_NAME=`basename ${0} .ksh`                                   # Name of the shell script which is executed
LCK_FILE="/tmp/${SCRIPT_NAME}_${2}.run"                            # Lock file/Temporary file to ensure that the files are not run multiple times
TMP_SFTP_SCRPT="/tmp/sftp_script_${2}_$$.ftp"                      # File used for build of SFTP tasks
DIR_NAME=`dirname ${0}`                                            # Directory from which shell script is executed
FILE_CONFIG="${DIR_NAME}/${1}.config"                              # File that defines the source and target specifications
FOUND=false
TIMESTAMP=`date '+%d-%b-%Y_%R'`
FILES_PATH="/tmp/Files2Transfer_${2}"  
ARCHIVE_PATH="/tmp/iBuyFilesTransfered/archive"

#
# Copy local files to remote file system
#
Local2Remote() {

	cd ${FILES_PATH} 2>/dev/null 
	if [[ ${?} -gt 0 ]];then 
          print -u2 "LocalTarget path not found";
          rm ${LCK_FILE}
          exit -1;
        fi

	echo cd ${FILE_TARGET_PATH} > ${TMP_SFTP_SCRPT} 
	echo mput ${FILE_MASK} >> ${TMP_SFTP_SCRPT}  
	echo bye >> ${TMP_SFTP_SCRPT}

	sftp -oPort=${FILE_TARGET_PORT} -b ${TMP_SFTP_SCRPT} ${FILE_TARGET_ACCOUNT} 2>/dev/null 1>&2 

	rm ${TMP_SFTP_SCRPT}

}


#
# Copy remote files to local file system
#
Remote2Local() {

	cd ${FILES_PATH} 2>/dev/null
	if [[ ${?} -gt 0 ]];then
          print -u2 "LocalSource path not found";
          rm ${LCK_FILE}
          exit 1;
        fi

	echo cd ${FILE_SOURCE_PATH} > ${TMP_SFTP_SCRPT}
	echo mget ${FILE_MASK} >> ${TMP_SFTP_SCRPT}
	# Changes added to archive the files
	# echo rm ${FILE_MASK} >> ${TMP_SFTP_SCRPT} Commented this line as the below zip command moves the files into archive and deletes the files from the system
	echo zip -m archive.zip ${FILE_MASK} >> ${TMP_SFTP_SCRPT} 
    # End of change
	echo bye >> ${TMP_SFTP_SCRPT}

	sftp -oPort=${FILE_SOURCE_PORT} -b ${TMP_SFTP_SCRPT} ${FILE_SOURCE_ACCOUNT} 2>/dev/null 1>&2 

	rm ${TMP_SFTP_SCRPT}
			
}

#
# Main function
#

if [ ! -f ${FILE_CONFIG} ]; then
	print -u2 "Configuration file not found"
	rm ${LCK_FILE}
        exit 1
fi

if [ -f ${LCK_FILE} ]; then
	print -u2 "Script with parameter ${CL_NAME} already running, exiting..."
	exit 1
else
	touch ${LCK_FILE}
fi

exec < ${FILE_CONFIG}
	while IFS=\| read -r name source_account source_port source_path target_account target_port target_path mask; do
	case ${name} in
		\#* | "")
			continue
			;;
	esac
		
			eval FILE_SOURCE_ACCOUNT=${source_account}
                        eval FILE_SOURCE_PORT=${source_port}
			eval FILE_SOURCE_PATH=${source_path}
			eval FILE_TARGET_ACCOUNT=${target_account}
                        eval FILE_TARGET_PORT=${target_port}
			eval FILE_TARGET_PATH=${target_path}
			eval FILE_MASK=${mask} 
			FOUND=true
			break

done

if [ ${FOUND} = false ]; then
	print -u2 "File to transfer not configured"
    rm ${LCK_FILE}
	exit 2
fi

mkdir ${FILES_PATH}

set -f
Remote2Local
Local2Remote
set +f

rm -Rf ${FILES_PATH}

rm ${LCK_FILE}

exit 0

Thanks in advance.

Regards,
Radhika.

Last edited by rbatte1; 06-17-2015 at 11:49 AM.. Reason: Changed ICODE tags to CODE tags
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Backing up or Archiving files in UNIX

Hi All, I am very new to the UNIX world and find myself in a new position at work that requires me to archive large CADD files based in both UNIX and Windows environments on CD's. I have one engineer that wants to export these files as a table (I guess) and it appears to have a lot of paper... (2 Replies)
Discussion started by: Dsartelle
2 Replies

2. Shell Programming and Scripting

Archiving and moving the files

hi all i have a requirement where in i have to zip all the files with "*.bkp" after 14 days and move the zip files to Archive directory .... i am able to achieve the first functionality but not able to achive the second one ...here is my code find ${LOG_DIR} -name "*.bkp" -mtime +14 | xargs -i... (1 Reply)
Discussion started by: nvuradi
1 Replies

3. Shell Programming and Scripting

Archiving the files

Hi, Suppose I have 2 files of yesterday's. And today I have received 3 files. Before processing anything I want to archieve the 2 files of yesterday's into a different folder. How can this be done? Regards, Sunitha (1 Reply)
Discussion started by: Sunitha_edi82
1 Replies

4. UNIX for Dummies Questions & Answers

Archiving and move files in the same time

Hi All, I have tried so many command but none work like i wanted. I would like archive which i assume it will move the files and archive it somewhere. for example: if i have a folder and files: /home/blah/test /home/blah/hello /home/blah/foo/bar i would like to archive folder... (6 Replies)
Discussion started by: c00kie88
6 Replies

5. Shell Programming and Scripting

Archiving the Files in a folder

My requirement is to put all the files from output directory(ATT) to archive directory(archive\) creating a new folder with datetimestamp(20100212_120014) every time it runs. where ${IMF_TARGET_DIR} is my base directory. ${IMF_ARCHIVE_DIR} is my Archive directory... (1 Reply)
Discussion started by: vsmeruga
1 Replies

6. Shell Programming and Scripting

Issue while archiving the files

Hi, In our current process we are reading the file, (which is placed by external vendor)from one particular folder and processing those files through ETL(informatica). We are reading these file as " ls -ltr *.txt" Once the process is finish these files are moved to archived script by "mv"... (1 Reply)
Discussion started by: Amey Joshi
1 Replies

7. UNIX for Dummies Questions & Answers

Archiving the files in a .txt file

HI , I have a file abc.txt, which has some .csv files listed. example. abc.txt 1.csv 2.csv 3.csv 4.csv 5.csv I want to move all the files listed in abc.txt to a archive directory,and zip the moved files. Can anyone help me with the script. Thanks,sai (1 Reply)
Discussion started by: saii
1 Replies

8. Shell Programming and Scripting

Shell script for log archiving

I have an nfs mount /logfile/project mounted on several of my application server machines. I have 5 jvms running on each machine and I have several machines. the jvms logs are created and rotated every day so it will look like /jvm1/logs/server.log.2010-10-27 /jvm2/logs/server.log.2010-10-27... (3 Replies)
Discussion started by: gubbu
3 Replies

9. Shell Programming and Scripting

Archiving older files

Hello Group, I would request your help to build a shell script in order to find files older than 90 days then create the same directory structure under the second disk (/archive directory) and move the file preserving the same timestamps (ownership, etc). Also keep the log of files moved... (4 Replies)
Discussion started by: csierra
4 Replies

10. Shell Programming and Scripting

Archiving the files

hi, Am trying to acrhive a bunch of files on some ftp site and somehow managed to come out with the below logic. I'm getting "syntax error: unexpected end of file" error. Interestingly this below snipeet works fine if run for the first time but the subsequent runs fail! Anybody has any idea... (3 Replies)
Discussion started by: Amee5
3 Replies
All times are GMT -4. The time now is 07:27 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy