I tried to automate moving files with the script below. I spent hours of research on the net creating what I have so far. Scripting does not come easy for me so any help would be greatly appreciated. Thanks in advance.
The files are date stamp (YYYYMMDD) in the file name when created. New files are created 6 days a week, Mon-Sat. I need to move a minimum of 6 files totaling 35GB.
The script checks df -k used space on two file systems. If used space exceeds thresholds then SCP to a file system on a different server. If SCP successful then rm files.
The SCP does not work in the script but I can run the command manually. The exit status returns a 0 consequently deleting the files. Not good.
The second "move" action did not work today. The script ended up deleting files. Not good, again.
Again I can run the second "move" event manually with no problems.
I swapped out sensitive names but for the most part the script is what it is. Thanks again for taking the time.
Code:
#!/bin/ksh
#Assign and export variables
UsedSpace_u06=0 ; export UsedSpace_u06
UsedSpace_u10=0 ; export UsedSpace_u10
FSu10=${1:-"/u10"} ; export FSu10
FSu6=${2:-"/u06"} ; export FSu6
Threshold_u10=${3:-58} ; export Threshold_u10
Threshold_u06=${4:-70} ; export Threshold_u06
U06_path=/u06/backup/oracle/XXXX/exports ; export U06_path
U10_path=/u10/backup/oracle/XXXX/exports ; export U10_path
LOG_path=/backup/home/oracle/logs ; export LOG_path
#df -k u10 and u06 file systems
UsedSpace_u10=`df -k $FSu10 | grep % | awk {'print $5'} | sed 's/%//g'`
UsedSpace_u06=`df -k $FSu6 | grep % | awk {'print $5'} | sed 's/%//g'`
#Create log file
echo "Running FS.sh to move files due to lack of disk space on `date`" > $LOG_path/FS_`date +%Y%m%d`.log
echo "/u06 capacity is" $UsedSpace_u06"%. " "/u10 capacity is" $UsedSpace_u10"%." >> $LOG_path/FS_`date +%Y%m%d`.log
#if fs > threshold find files to move to different filesystem and log event
if [ $UsedSpace_u06 -gt $Threshold_u06 ]
then
find $LOG_path -name "FS_*.log" -mtime 3 -exec rm '{}' +
echo "" >> $LOG_path/FS_`date +%Y%m%d`.log
echo "Files to be copied to XXXXX..." >> $LOG_path/FS_`date +%Y%m%d`.log
find $U06_path -name "expdp*.dmp.gz" -mtime +8 -exec ls '{}' >> $LOG_path/FS_`date +%Y%m%d`.log +
find $U06_path -name "expdp*.dmp.gz" -mtime +8 -exec scp -rp {} <HOSTNAME>:/u10/backup/oracle/BOCK/exports \;
fi
#if SCP ok delete files and log event
if [ "$?" -ne 0 ]
then
echo "" >> $LOG_path/FS_`date +%Y%m%d`.log
echo "SCP exit status is" "$?" >> $LOG_path/FS_`date +%Y%m%d`.log
echo "" >> $LOG_path/FS_`date +%Y%m%d`.log
else
echo "" >> $LOG_path/FS_`date +%Y%m%d`.log
echo "SCP exit status is" "$?" >> $LOG_path/FS_`date +%Y%m%d`.log \;
echo "" >> $LOG_path/FS_`date +%Y%m%d`.log
echo "Files to be deleted to /u06..." >> $LOG_path/FS_`date +%Y%m%d`.log
find $U06_path -name "expdp*.dmp.gz" -mtime +8 -exec ls '{}' >> $LOG_path/FS_`date +%Y%m%d`.log +
find $U06_path -name "expdp*.dmp.gz" -mtime +8 -exec rm '{}' +
fi
#if fs > threshold find files to move and log event
if [ $UsedSpace_u10 -gt $Threshold_u10 ]
then
echo "Files to be moved to /u06..." >> $U06_path/FS_`date +%Y%m%d`.log
find $U10_path -name "expdp*.dmp.gz" -mtime +4 -exec ls '{}' >> $LOG_path/FS_`date +%Y%m%d`.log +
find $U10_path -name "expdp*.dmp.gz" -mtime +4 -exec mv '{}' $U06_path +
echo "" >> $LOG_path/FS_`date +%Y%m%d`.log
fi
#log event
echo "Exit Status for U10 move is" "$?" >> $LOG_path/FS_`date +%Y%m%d`.log \;
echo "Script completed on `date`." >> $LOG_path/FS_`date +%Y%m%d`.log \;
I would suggest to move the scp to a while loop and check the return code. I would do the same for the other operations, rm, mv, etc... This way you can handle the error in a graceful way and log which operation/file failed. An alternative to this would be to set a trap but this might not cover all situations.
ie:
Code:
find .... | while read file
do
scp $file ...
if [[ $? -ne 0 ]]
then
email and exit with error
fi
done
Thanks Frank for the suggestion. It made the difference. I added 2 while loops for the scp and mv events. Within the scp loop I put an if-then-else statement to check the exit status. If true rm the $file. One of the other things I forgot to do is add PATH statement. Thanks again for your help. I am one step closer getting my arms around this scripting stuff.
i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B ,... (2 Replies)
Hello,
First time poster. I am looking for a way to script or program the process of moving files from one folder to another, automatically, based on the count of files in the destination folder.
I was thinking a shell script would work, but am open to the suggestions of the experts... (6 Replies)
Hey guys,
I have wrote the following script to apply a module named "trinity" on my files. (it takes two input files and spit a trinity.fasta as output)
#!/bin/bash -l
#SBATCH -p node
#SBATCH -A <projectID>
#SBATCH -n 16
#SBATCH -t 7-00:00:00
#SBATCH --mem=128GB
#SBATCH --mail-type=ALL... (1 Reply)
I want to move all files from one directory to another directory excluding today (sysdate files) on daily basis.
file name is in pattern file_2013031801, file_2013031802 etc (2 Replies)
I have some files that want to be copy to remote server , and then verify if all files are successfuly copy , if yes, then remove it , can advise how to write a script ?
thanks .
---------- Post updated 11-29-12 at 06:11 AM ---------- Previous update was 11-28-12 at 11:16 PM ----------
... (2 Replies)
I would like to transfer all files ending with .log from /tmp and to /tmp/archive (using find )
The directory structure looks like :-
/tmp
a.log
b.log
c.log
/abcd
d.log
e.log
When I tried the following command , it movies all the log files... (8 Replies)
I know I can use an ls -l junk1 command to get a listing of all files in the directory junk1, but I was wondering how I'd go about going through the files in junk1 in a for-in loop and issuing the ls -l command on them one by one.
This is what I have so far:
for file in $(ls -a $1)
do
ls... (1 Reply)
I have a ksh script that currently moves a day's worth of log files (about 15,000) files to a different directory. The issue is that about 100 of these files are still open for write when this happens. I need an efficient way to ensure that these files aren't open without doing an lsof on each... (7 Replies)
Hi,
I am a newbie to shell scripting. here is my objective:
1)The shell program should take 2 parameters - ie-> DestinationFolder, WebFolder
2)Destination folder contains few files that has to has be verified and deleted.
3)WebFolder is a folder containing a list of master files
4)It... (1 Reply)
