Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Archiving files using shell script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Archiving files using shell script
# 1  
Old 06-17-2015
Archiving files using shell script
Dear Team,



I am looking for transferring files to and from the local and remote servers using SFTP commands. Currently the script is using the mget and mput commands to do the copying of the files. While I am trying to move the files from local to remote server, I would also like to archive or save a copy in the local system. Could someone suggest me the suitable commands for the same?
Currently the script is like this:
Code:
---Copy remote files to local file system

echo cd ${FILE_SOURCE_PATH} > ${TMP_SFTP_SCRPT}

echo mget ${FILE_MASK} >> ${TMP_SFTP_SCRPT}

echo rm ${FILE_MASK} >> ${TMP_SFTP_SCRPT}  ## this has to be modified to archive the files to the local system directory and then remove the file##

echo bye >> ${TMP_SFTP_SCRPT}

sftp -oPort=${FILE_SOURCE_PORT} -b ${TMP_SFTP_SCRPT} ${FILE_SOURCE_ACCOUNT} 2>/dev/null 1>&2


Thanks in advance for your help.
Last edited by rbatte1; 06-17-2015 at 10:49 AM.. Reason: Added CODE tags
# 2  
Old 06-17-2015
Heyas Rads

Please use code tags (not none / not icode) for multiline code examples.

When you download a file to local, using mget its already copied local, so to archive it localy, you would have to finish the sftp sessions, and then do a copy of that file you just downloaded.

Obviously most of your commands dont work, as you have put an [ICODE]echo[/ICODE] in front of them.
I suggest to remove all those [ICODE]echo[/ICODE]'s, but leave the [ICODE]echo rm ...[/ICODE] so the remove files do not get deleted (just yet).

Otherwise, your code looks 'ok'.
This said, i dont really understand your question, could yuu please elabrote a bit more?


hth
This User Gave Thanks to sea For This Post:
Rads
# 3  
Old 06-17-2015
Firstly I would streamline the code a little. You open the file quite a few times. You can easily do this:-
Code:
{
   echo "First line"
   echo "Second line"
} > filename

If you are driving commands based on what is on a remove server, then you may have to break this into several sftp steps:-
  • Get list of files on remote server
  • Loop for each file
    • SFTP get the file
    • Confirm file okay
    • SFTP delete the file
    • Confirm delete okay (perhaps try to list the file with SFTP)
  • Report outcome of the overall job
Does thinking about breaking up your overall process with this logic help?



Robin
# 4  
Old 06-18-2015
Hi Robin,

Thanks for the information. I am new to shell scripting and this script for archiving is a bit challenging for me. Your help would be much appreciated. Please find the code as below:
Code:
#!/bin/ksh
# Purpose                      : Transfer files from Local to Remote/Remote to 
#Name of the file that is executed
SCRIPT_NAME=`basename ${0} .ksh`    
#Lock file/Temporary file to ensure that the files are not run multiple times                                         
LCK_FILE="/tmp/${SCRIPT_NAME}_${2}.run"  
#File used for build of SFTP tasks                                    
TMP_SFTP_SCRPT="/tmp/sftp_script_${2}_$$.ftp"  
#Denotes the current directory                              
DIR_NAME=`dirname ${0}`    
#File that defines the source and target specifications                                                  
FILE_CONFIG="${DIR_NAME}/${1}.config"                                        
FOUND=false
TIMESTAMP=`date '+%d-%b-%Y_%R'`
FILES_PATH="/tmp/Files2Transfer_${2}"

#
# Copy files from local system to remote system
#
Local2Remote() {

	cd ${FILES_PATH} 2>/dev/null
	if [[ ${?} -gt 0 ]];then
          print -u2 "LocalTarget path not found";
          rm ${LCK_FILE}
          exit -1;
        fi

	echo cd ${FILE_TARGET_PATH} > ${TMP_SFTP_SCRPT}
	echo mput ${FILE_MASK} >> ${TMP_SFTP_SCRPT}
	echo bye >> ${TMP_SFTP_SCRPT}

	sftp -oPort=${FILE_TARGET_PORT} -b ${TMP_SFTP_SCRPT} ${FILE_TARGET_ACCOUNT} 2>/dev/null 1>&2

	rm ${TMP_SFTP_SCRPT}

}

#
# Copy files from remote system to local system
#
Remote2Local() {

	cd ${FILES_PATH} 2>/dev/null
	if [[ ${?} -gt 0 ]];then
          print -u2 "LocalSource path not found";
          rm ${LCK_FILE}
          exit 1;
        fi

	echo cd ${FILE_SOURCE_PATH} > ${TMP_SFTP_SCRPT}
	echo mget ${FILE_MASK} >> ${TMP_SFTP_SCRPT}
	
# The below line needs to be modified to enable archive functionality. 	
  echo rm ${FILE_MASK} >> ${TMP_SFTP_SCRPT}    

# Added below line to copy and remove the files to the desired folder 
# echo mv ${file_mask} >> ${tmp_sftp_scrpt}                                  	
		          
# End of change 
	echo bye >> ${TMP_SFTP_SCRPT}

	sftp -oPort=${FILE_SOURCE_PORT} -b ${TMP_SFTP_SCRPT} ${FILE_SOURCE_ACCOUNT} 2>/dev/null 1>&2

	rm ${TMP_SFTP_SCRPT}
			
}

#
# Main function
#

if [ ! -f ${FILE_CONFIG} ]; then
	print -u2 "Configuration file not found"
	rm ${LCK_FILE}
        exit 1
fi

if [ -f ${LCK_FILE} ]; then
	print -u2 "Script with parameter ${CL_NAME} already running, exiting..."
	exit 1
else
	touch ${LCK_FILE}
fi

exec < ${FILE_CONFIG}
	while IFS=\| read -r name source_account source_port source_path target_account target_port target_path mask; do
	case ${name} in
		\#* | "")
			continue
			;;
	esac
		
			eval FILE_SOURCE_ACCOUNT=${source_account}
                        eval FILE_SOURCE_PORT=${source_port}
			eval FILE_SOURCE_PATH=${source_path}
			eval FILE_TARGET_ACCOUNT=${target_account}
                        eval FILE_TARGET_PORT=${target_port}
			eval FILE_TARGET_PATH=${target_path}
			eval FILE_MASK=${mask} 
			FOUND=true
			break

done

if [ ${FOUND} = false ]; then
	print -u2 "File to transfer not configured"
    rm ${LCK_FILE}
	exit 2
fi

mkdir ${FILES_PATH}

set -f
Remote2Local
Local2Remote
set +f

rm -Rf ${FILES_PATH}

rm ${LCK_FILE}

exit 0

I am following below process to test the files:
This script and the config file exists in one server. I am creating source and target folder in FTP server and putting in a sample CSV in source folder. I then login via putty and execute this script name.ksh<space> "config file name". Upon execution, I should be able to see the CSV file in the target folder.

I would like to know if I am using the correct method to do the archival process. Also is it possible to do the logging during the archival process (i.e., between archive start and end) to include name, timestamp information?

---------- Post updated 06-18-15 at 11:11 AM ---------- Previous update was 06-17-15 at 04:07 PM ----------

Hello,

I have now modified the script to include a functionality to archive the files. Please find the code as below. Could somebody suggest how I can log the information (start time of archival, files archived, folder to which files are moved, archival end time) in the script?
Code:
CL_NAME=${1}
SCRIPT_NAME=`basename ${0} .ksh`                                   # Name of the shell script which is executed
LCK_FILE="/tmp/${SCRIPT_NAME}_${2}.run"                            # Lock file/Temporary file to ensure that the files are not run multiple times
TMP_SFTP_SCRPT="/tmp/sftp_script_${2}_$$.ftp"                      # File used for build of SFTP tasks
DIR_NAME=`dirname ${0}`                                            # Directory from which shell script is executed
FILE_CONFIG="${DIR_NAME}/${1}.config"                              # File that defines the source and target specifications
FOUND=false
TIMESTAMP=`date '+%d-%b-%Y_%R'`
FILES_PATH="/tmp/Files2Transfer_${2}"  
ARCHIVE_PATH="/tmp/iBuyFilesTransfered/archive"

#
# Copy local files to remote file system
#
Local2Remote() {

	cd ${FILES_PATH} 2>/dev/null 
	if [[ ${?} -gt 0 ]];then 
          print -u2 "LocalTarget path not found";
          rm ${LCK_FILE}
          exit -1;
        fi

	echo cd ${FILE_TARGET_PATH} > ${TMP_SFTP_SCRPT} 
	echo mput ${FILE_MASK} >> ${TMP_SFTP_SCRPT}  
	echo bye >> ${TMP_SFTP_SCRPT}

	sftp -oPort=${FILE_TARGET_PORT} -b ${TMP_SFTP_SCRPT} ${FILE_TARGET_ACCOUNT} 2>/dev/null 1>&2 

	rm ${TMP_SFTP_SCRPT}

}


#
# Copy remote files to local file system
#
Remote2Local() {

	cd ${FILES_PATH} 2>/dev/null
	if [[ ${?} -gt 0 ]];then
          print -u2 "LocalSource path not found";
          rm ${LCK_FILE}
          exit 1;
        fi

	echo cd ${FILE_SOURCE_PATH} > ${TMP_SFTP_SCRPT}
	echo mget ${FILE_MASK} >> ${TMP_SFTP_SCRPT}
	# Changes added to archive the files
	# echo rm ${FILE_MASK} >> ${TMP_SFTP_SCRPT} Commented this line as the below zip command moves the files into archive and deletes the files from the system
	echo zip -m archive.zip ${FILE_MASK} >> ${TMP_SFTP_SCRPT} 
    # End of change
	echo bye >> ${TMP_SFTP_SCRPT}

	sftp -oPort=${FILE_SOURCE_PORT} -b ${TMP_SFTP_SCRPT} ${FILE_SOURCE_ACCOUNT} 2>/dev/null 1>&2 

	rm ${TMP_SFTP_SCRPT}
			
}

#
# Main function
#

if [ ! -f ${FILE_CONFIG} ]; then
	print -u2 "Configuration file not found"
	rm ${LCK_FILE}
        exit 1
fi

if [ -f ${LCK_FILE} ]; then
	print -u2 "Script with parameter ${CL_NAME} already running, exiting..."
	exit 1
else
	touch ${LCK_FILE}
fi

exec < ${FILE_CONFIG}
	while IFS=\| read -r name source_account source_port source_path target_account target_port target_path mask; do
	case ${name} in
		\#* | "")
			continue
			;;
	esac
		
			eval FILE_SOURCE_ACCOUNT=${source_account}
                        eval FILE_SOURCE_PORT=${source_port}
			eval FILE_SOURCE_PATH=${source_path}
			eval FILE_TARGET_ACCOUNT=${target_account}
                        eval FILE_TARGET_PORT=${target_port}
			eval FILE_TARGET_PATH=${target_path}
			eval FILE_MASK=${mask} 
			FOUND=true
			break

done

if [ ${FOUND} = false ]; then
	print -u2 "File to transfer not configured"
    rm ${LCK_FILE}
	exit 2
fi

mkdir ${FILES_PATH}

set -f
Remote2Local
Local2Remote
set +f

rm -Rf ${FILES_PATH}

rm ${LCK_FILE}

exit 0

Thanks in advance.

Regards,
Radhika.
Last edited by rbatte1; 06-17-2015 at 11:49 AM.. Reason: Changed ICODE tags to CODE tags
# 5  
Old 06-18-2015
Quote:
Originally Posted by Rads
Thanks for the information. I am new to shell scripting and this script for archiving is a bit challenging for me.
Before i start giving you tips on how to improve your script there is a general point to discuss:

Archiving files is usually done with the intent of being able to restore some previous state of a system (at least locally, say, in a certain directory). This means you do not only have to restore a files (directories) contents but some meta-information with it. Let us look at a certain file:

Code:
# ls -lai /some/directory
total 2832
40960 drwxr-xr-x    5 root     system         4096 May 27 15:09 .
    2 drwxr-xr-x   31 bin      bin            4096 Apr 27 13:38 ..
40961 -rw-r--r--    1 myuser   mygroup        1333 Oct  7 2013  somefile


Obviously there is the content of the file - the 1333 bytes it takes on the disk. There is also the file mode ("rw-r--r--" or "644"), there is the ownership (owner=myuser, group membership="mygroup"), the inode number (40961) and (not completely visible) three time stamps: the creation date/time, the last modification date/time and the last access date/time. Anything beyond the file content is stored in the inode of the file.

You may need some or even all of these metadata to go along with the raw content to form an archive enabling you to restore the file.

This is why there are special archiving programs (namely "tar", "cpio" and their successor "pax") which do exactly that. This is why you should consider creating tar- (or cpio-, pax-, ...) archives first and only as a last step transfer these archives to remote systems. This way you do not need to meddle with "mget" and other options of "ftp". A single "get <archive.file>" would suffice.

I hope this helps.

bakunin
# 6  
Old 06-18-2015
Hi,

Thanks for the detailed information and the tips.

I am completely new to this and it would be great help if you can suggest me the suitable changes to be made to the script for archiving the files and also for logging the information.

My peers are not recommending the usage of TAR or other functionalities.

Regards,
Radhika.
# 7  
Old 06-18-2015
You need to define clearly the aims, then break them down into smaller tasks. Try to draw the process or write it out. Keep clear that the three basic logical flow types are:-
  • Sequence
  • Branch
  • Loop
The sequence can be any number of statements (including the other two) and calls to commands etc.


Is there a reason your peers are 'not recommending' tar? Do they not trust it, or is it that they just haven't suggested it.

It is nice that your script uses functions (Local2Remote etc.) but as they are called just once, it seems a little odd. Perhaps it makes the main part of the scripts easier to read, but I also worry that you are trying to call zip within sftp and I'm not sure it will work.

I would suggest that your two test at the beginning of the main part are the wrong way round. You should test for the lock-file first, then the configuration file.

I'm not too sure on why you exec < ${FILE_CONFIG} in your code the read in in your loop. Could you just simply . ${FILE_CONFIG} which will execute the file in the current shell, i.e. any variables or functions set are then available to the script that called it.

Your loop with a break and the eval statements are a little perplexing too.

Can you explain what your input file FILE_CONFIG will contain and the eventual process you want to achieve? There may be a simpler way to get there.


There is definitely some good stuff in the script though, so don't lose confidence. I'm just trying to understand the overall objective and trying to get the best outcome for you in a way that you can maintain.



Robin
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Archiving the files

hi, Am trying to acrhive a bunch of files on some ftp site and somehow managed to come out with the below logic. I'm getting "syntax error: unexpected end of file" error. Interestingly this below snipeet works fine if run for the first time but the subsequent runs fail! Anybody has any idea... (3 Replies)
Discussion started by: Amee5
3 Replies

2. Shell Programming and Scripting

Archiving older files

Hello Group, I would request your help to build a shell script in order to find files older than 90 days then create the same directory structure under the second disk (/archive directory) and move the file preserving the same timestamps (ownership, etc). Also keep the log of files moved... (4 Replies)
Discussion started by: csierra
4 Replies

3. Shell Programming and Scripting

Shell script for log archiving

I have an nfs mount /logfile/project mounted on several of my application server machines. I have 5 jvms running on each machine and I have several machines. the jvms logs are created and rotated every day so it will look like /jvm1/logs/server.log.2010-10-27 /jvm2/logs/server.log.2010-10-27... (3 Replies)
Discussion started by: gubbu
3 Replies

4. UNIX for Dummies Questions & Answers

Archiving the files in a .txt file

HI , I have a file abc.txt, which has some .csv files listed. example. abc.txt 1.csv 2.csv 3.csv 4.csv 5.csv I want to move all the files listed in abc.txt to a archive directory,and zip the moved files. Can anyone help me with the script. Thanks,sai (1 Reply)
Discussion started by: saii
1 Replies

5. Shell Programming and Scripting

Issue while archiving the files

Hi, In our current process we are reading the file, (which is placed by external vendor)from one particular folder and processing those files through ETL(informatica). We are reading these file as " ls -ltr *.txt" Once the process is finish these files are moved to archived script by "mv"... (1 Reply)
Discussion started by: Amey Joshi
1 Replies

6. Shell Programming and Scripting

Archiving the Files in a folder

My requirement is to put all the files from output directory(ATT) to archive directory(archive\) creating a new folder with datetimestamp(20100212_120014) every time it runs. where ${IMF_TARGET_DIR} is my base directory. ${IMF_ARCHIVE_DIR} is my Archive directory... (1 Reply)
Discussion started by: vsmeruga
1 Replies

7. UNIX for Dummies Questions & Answers

Archiving and move files in the same time

Hi All, I have tried so many command but none work like i wanted. I would like archive which i assume it will move the files and archive it somewhere. for example: if i have a folder and files: /home/blah/test /home/blah/hello /home/blah/foo/bar i would like to archive folder... (6 Replies)
Discussion started by: c00kie88
6 Replies

8. Shell Programming and Scripting

Archiving the files

Hi, Suppose I have 2 files of yesterday's. And today I have received 3 files. Before processing anything I want to archieve the 2 files of yesterday's into a different folder. How can this be done? Regards, Sunitha (1 Reply)
Discussion started by: Sunitha_edi82
1 Replies

9. Shell Programming and Scripting

Archiving and moving the files

hi all i have a requirement where in i have to zip all the files with "*.bkp" after 14 days and move the zip files to Archive directory .... i am able to achieve the first functionality but not able to achive the second one ...here is my code find ${LOG_DIR} -name "*.bkp" -mtime +14 | xargs -i... (1 Reply)
Discussion started by: nvuradi
1 Replies

10. UNIX for Dummies Questions & Answers

Backing up or Archiving files in UNIX

Hi All, I am very new to the UNIX world and find myself in a new position at work that requires me to archive large CADD files based in both UNIX and Windows environments on CD's. I have one engineer that wants to export these files as a table (I guess) and it appears to have a lot of paper... (2 Replies)
Discussion started by: Dsartelle
2 Replies
Login or Register to Ask a Question