Ok so once again im back with what is probably a beginner question although somewhat more complicated (for me) than the last.
Background:
A client has a daily backup which is carried out via rsync.
Due to this, when they move a file around that file is then coppied a second time.
On top of this, when a file is deleted off of the client server it isnt deleted off of the backup server. (Rsync options i believe)
Im am working on a script that does the following:
Entities:
1. Client File Server (A)
2. Offsite Backup Server (B)
Current:
1. Backup sync runs daily.
2. Changed / new files on A in comparison to B are replaced / copied onto B.
3. Deleted files on A are ignored in this process
Proposed:
Additional script created to do the following:
1. Created a file of all files currently on A.
2. Compare with a file of all files currently on B.
3. If:
a) A has a file B doesn't < Warning report on email
b) B has a file A doesn't < Mark for deletion
c) A has a modified file to B < Warning report on email
4. Do the following:
-i < Identify / Create a report with what would have been deleted
5. If no restriction is placed then delete.
Thoughts:
1. Rsync with option delete files on B that are no longer on A
2. Personal script.
Methods within script:
Main
< Call method
Creation
< Create A file
< Create B file
Comparison
< Check for discrepancies between files
< Create file with discrepancies
Output
< Send error messages to unison email account
< Display files to be deleted on screen then delete
Outputed Email:
1. No. files processed
2. List of deleted files + size for deletion
3. Total deleted size
4. Errors/Warnings
< Email sent regardless of parameter selected or not.
So far this is what i have:
Code:
#!/bin/sh
#Initalise Variables
clientDIR=0
backupDIR=0
clientFile=0
backupFile=0
dateStamp=0
saveLocation=0
#1. Create client file:
clientDIR=/home/[USR]/
dateStamp=`date +%d-%m-%Y`
saveLocation=/home/root/scripts
clientFile=$saveLocation/clientFile@$dateStamp.txt
cd $clientDIR
for i in "$clientDIR"
do
ls -lstR > $clientFile
done
#2. Create backup file:
#ssh to offsite backup location using key rather than pw
#backupFile=insert ssh login without pw using key to ip + /backupFile@$dateStamp.txt
#TEMP TEST FILE
backupFile=$saveLocation/backupFile@$dateStamp.txt
cd $clientDIR
for i in "$clientDIR"
do
ls -lstR > $backupFile
done
#3: To compare files:
#cmp <- Compare two files
#Not working
#comm <- Compare two sorted files line by line
#Not working
#diff <- Display the differences between two files
diff -y $clientFile $backupFile > $saveLocation/diffOutput@$dateStamp.txt
#diff command doesnt recognise files due to order....
#4: If:
# a) A has a file B doesn't < Warning report
# b) B has a file A doesn't < Mark for deletion
# c) A has a modified file to B < Warning report
#5. Do the following:
# -i < Identify / Create a report with what would have been deleted
#6. If no restriction is placed then delete.
echo "Done <- Debugging"
So what i need from you guys will be a fair bit of help (i think) but if we can start with how i can compare the files generated. ATM with the diff command due to files being in a different order with the backup server having more files than the client, naturally the file lines wont match up.
Is there an easier way to do this? Any help is appreciated. Feel free to tell me im going the wrong way about this and to try something else.
Many regards and sorry to any that read this super long post,
Regards,
John Crombie
I think all the tasks you want to achieve can be done using rsync options only.
No need to write down a separate script for that.
rsync has option for dry run, i.e. instead of syncing data just compare files and provide list of files (building file list... o/p) which are going to be sync in process.
As you mentioned, rsync has option for deleting files which are on A and not in B.
The only thing is if you want to check which files are going to be deleted from B before getting deleted. For that can you check running dry run with --delete option and check what o/p comes?
One better approach is to setup backup policy as
Delete/Archive everything on B on Friday night, depending upon disk space.
Take full backup on Sat/sun.
Run incremental backup everyday till next Sat/Sun.
I know, you would have more queries, most welcome.
Ok so i took into account what you siad about rsync and managed something but now i want to finish what i started...after redefining what i need to do to myself i came to the following:
This is what i need processFolder() to do :P
Code:
#process folder(folder) {
# for all the entreies in this folder
# if current_thing = file
# if file existis on client
# if size is no same
# flag error
# else
# OK
# else
# flag for deletion
# elif current_thing = dir
# process folder(current_thing)
# elif
# error - not file or folder
# fi
#}
And this is what i currently have....
Code:
#!/bin/sh
processFolder() {
while ($BF_VAL="1") do
if [ -f $BF_VAL ]
then
echo "file"
else
echo "dir"
fi
}
echo "Initilising variables"
#CF = Client File
#BS = Backup Folder
#D = Dry-run
CF_ARG=0
CF_VAL=
BF_ARG=0
BF_VAL=
D_ARG=0
parseArgs()
{
while [ $# -gt 0 ]
do
case "$1" in
-d) D_ARG=1;;
-c) CF_ARG=1; CF_VAL=$2; shift;;
-b) BF_ARG=1; BF_VAL=$2; shift;;
?) printHelp;;
*) echo "$1 not an option"; printHelp;;
esac
shift
done
}
printHelp()
{
echo "Usage: ./test-v2.sh -d -c [filename] -b [folder location]"
echo ""
echo "-d = Dry-run."
echo " ie. Run through the program as normal but only output a file with
the changes that could be made."
echo ""
echo "-c = Client File input."
echo " ie. Supply the location and name of the previously generated client
file list."
echo ""
echo "-b = Backup Folder Location."
echo " ie. Supply the location of where the backup is currently stored."
}
# main starts here
parseArgs $*
echo "Reading client file " $CF_VAL
echo "Reading backup folder " $BF_VAL
processFolder $BF_VAL
Any furthur input on this? ASAP but i know you all have lives
Ok so i have made some progress but again im stumped... this is what i have so far:
Code:
#!/bin/sh
#Version 3.1
#Program to rationalise the off site backup in comparison to the current client file system.
processFolder() {
if !(test -d "$1") then
echo "do stuff with file"
# Use cases for file checking:
# a) SRC has a file DEST doesn't < Warning report
# b) DEST has a file SRC doesn't < Mark for deletion
# c) SRC has a bigger size file to DEST < Warning report
# d) DEST has a bigger file size to SRC < Warning report
#* e) SRC has a diff date file to DEST < Warning report
# f) DEST has a directory SRC doesn't < Mark for deletion
# Pesudo Code for file checking:
# if exits on SRC = yes then
# if same size = yes then
# process_folder
# if same size = no then #c), d)
# flag warning
# if exist on SRC = no then #b)
# mark for deletion
#
#elif item = directory then
# if exists on SRC = yes then
# process_folder
# if exists on SRC = no then #f)
# mark for deletion
#
#else
# echo "'current item' Is not file or directory"
# Existence checking
# Need help here:
# Does file exist on client server?
# Size Checking
# BAKS contains size of file1
# CLNTS contains size of file2
BAKS=$(stat -c%s "$1")
CLNTS=$(stat -c%s "$2")
if [[ "$BAKS" -lt "$CLNTS" ]]; then
echo "Bbackup file is smaller"
echo "Warning, backup not successful"
elif [[ "$BAKS" -gt "$CLNTS" ]]; then
echo "Client file size is smaller"
echo "Warning, backup not successful"
elif [[ "$BAKS" -eq "$CLNTS" ]]; then
processFolder ()
fi
else
cd $1
for i in *
do
processFolder "$i"
done
fi
}
parseArgs()
{
if [ $# -lt 4 -o $# -gt 5 ]; then
printHelp
return 1;
else
while [ $# -gt 0 ]
do
case "$1" in
-d) D_ARG=1;;
-c) CF_ARG=1; CF_VAL=$2; shift;;
-b) BF_ARG=1; BF_VAL=$2; shift;;
?) printHelp;;
*) echo "$1 not an option"; printHelp;;
esac
shift
done
fi
}
printHelp()
{
echo "Usage: $0 -d -c [filename] -b [folder location]"
echo ""
echo "-d = Dry-run."
echo " ie. Run through the program as normal but only output a file with the changes that could be made."
echo ""
echo "-c = Client File input."
echo " ie. Supply the location and name of the previously generated client file list."
echo ""
echo "-b = Backup Folder Location."
echo " ie. Supply the location of where the backup is currently stored."
}
#main starts here
echo "Welcome to version 3.1"
echo "Initilising variables"
echo ""
#CF = Client File
#BS = Backup Folder
#D = Dry-run
CF_ARG=0
CF_VAL=
BF_ARG=0
BF_VAL=
D_ARG=0
parseArgs $*
PARGS=$?
if [ $PARGS -eq 0 ]; then
echo "Reading client file " $CF_VAL
echo "Reading backup folder " $BF_VAL
processFolder $BF_VAL
return $?
else
return $PARGS
fi
So what im aiming to do is the commening at the top just inside of processFolder ()
Hi all
I have a unix based firewall, which creates a daily backup file on the device.
I need a script to scp this file over to a remote server.
I can get this working daily using a basic script and a cron job.
However, I only want it to send the latest config back up file and currently... (4 Replies)
We are taking backup of our application data(cobol file system, AIX/unix) before and after EOD job runs. The data size is approximately 260 GB in biggest branch. To reduce the backup time, 5 parallel execution is scheduled through control-m which backups up the files in 5 different *.gz. The job... (8 Replies)
We are taking backup of our application data(cobol file system, AIX/unix) before and after EOD job runs. The data size is approximately 260 GB in biggest branch. To reduce the backup time, 5 parallel execution is scheduled through control-m which backups up the files in 5 different *.gz. The job... (2 Replies)
HI all, im new to shell scripting. need your guidence for my script. i wrote one script and is attached here
Im explaining the requirement of script.
AIM: Shell script to run automatically as per scheduled and backup few network devices configurations. Script will contain a set of commands... (4 Replies)
Hi Everyone,
we are running rsync with --backup mode, Are there any rsync options to remove backup folders on successful deployment?
Thanks in adv. (0 Replies)
Hi Experts,
I am very beginner in Bash Shell Scripting. Can anyone please guide me to create a script which should show the most busy file systems in sort basis as there are a lot of file systems on the server.
I was told this task to be done by my IT lead and I must have to do this in... (3 Replies)
My data is something like shown below.
date1 date2 aaa bbbb ccccc
date3 date4 dddd eeeeeee ffffffffff ggggg hh
I want the output like this
date1date2 aaa eeeeee
I serached in the forum but didn't find the exact matching solution. Please help. (7 Replies)
The line is like this
+abc+def+mgh+ddsdsd+sa
i.e. words seperated by +. There is a plus in the beginning.
i want to conver this line to
abc, def, mgh, ddsdsd, sa
please provide the logic in the form of a shell script
Thanks in advance (13 Replies)
Hi there,
i am trying to create a script that checks for the existence of users on the system, if they exist then their details should print on the screen with a message that the id is in use. I am having a bit of trouble with it.
Any ideas?
Cheers
Kev! (8 Replies)