Backup Rationalisation Script - Help Required


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Backup Rationalisation Script - Help Required
# 1  
Old 08-30-2011
Backup Rationalisation Script - Help Required

Ok so once again im back with what is probably a beginner question although somewhat more complicated (for me) than the last.

Background:
A client has a daily backup which is carried out via rsync.
Due to this, when they move a file around that file is then coppied a second time.
On top of this, when a file is deleted off of the client server it isnt deleted off of the backup server. (Rsync options i believe)

Im am working on a script that does the following:

Entities:
1. Client File Server (A)
2. Offsite Backup Server (B)

Current:
1. Backup sync runs daily.
2. Changed / new files on A in comparison to B are replaced / copied onto B.
3. Deleted files on A are ignored in this process

Proposed:
Additional script created to do the following:
1. Created a file of all files currently on A.
2. Compare with a file of all files currently on B.
3. If:
a) A has a file B doesn't < Warning report on email
b) B has a file A doesn't < Mark for deletion
c) A has a modified file to B < Warning report on email
4. Do the following:
-i < Identify / Create a report with what would have been deleted
5. If no restriction is placed then delete.

Thoughts:
1. Rsync with option delete files on B that are no longer on A

2. Personal script.
Methods within script:
Main
< Call method

Creation
< Create A file
< Create B file

Comparison
< Check for discrepancies between files
< Create file with discrepancies

Output
< Send error messages to unison email account
< Display files to be deleted on screen then delete

Outputed Email:
1. No. files processed
2. List of deleted files + size for deletion
3. Total deleted size
4. Errors/Warnings

< Email sent regardless of parameter selected or not.

So far this is what i have:
Code:
#!/bin/sh

#Initalise Variables
clientDIR=0
backupDIR=0
clientFile=0
backupFile=0
dateStamp=0
saveLocation=0

#1. Create client file:
clientDIR=/home/[USR]/
dateStamp=`date +%d-%m-%Y`
saveLocation=/home/root/scripts
clientFile=$saveLocation/clientFile@$dateStamp.txt

cd $clientDIR
for i in "$clientDIR"
do
       ls -lstR > $clientFile
done

#2. Create backup file:
#ssh to offsite backup location using key rather than pw
#backupFile=insert ssh login without pw using key to ip + /backupFile@$dateStamp.txt

#TEMP TEST FILE
backupFile=$saveLocation/backupFile@$dateStamp.txt

cd $clientDIR
for i in "$clientDIR"
do
        ls -lstR > $backupFile
done

#3: To compare files:
#cmp <- Compare two files
#Not working

#comm <- Compare two sorted files line by line
#Not working

#diff <- Display the differences between two files
diff -y $clientFile $backupFile > $saveLocation/diffOutput@$dateStamp.txt
#diff command doesnt recognise files due to order....

#4: If:
#       a) A has a file B doesn't < Warning report
#       b) B has a file A doesn't < Mark for deletion
#       c) A has a modified file to B < Warning report

#5. Do the following:
#       -i < Identify / Create a report with what would have been deleted

#6. If no restriction is placed then delete.

echo "Done <- Debugging"

So what i need from you guys will be a fair bit of help (i think) but if we can start with how i can compare the files generated. ATM with the diff command due to files being in a different order with the backup server having more files than the client, naturally the file lines wont match up.

Is there an easier way to do this? Any help is appreciated. Feel free to tell me im going the wrong way about this and to try something else.

Many regards and sorry to any that read this super long post,
Regards,
John Crombie
# 2  
Old 08-30-2011
I think all the tasks you want to achieve can be done using rsync options only.
No need to write down a separate script for that.

rsync has option for dry run, i.e. instead of syncing data just compare files and provide list of files (building file list... o/p) which are going to be sync in process.

As you mentioned, rsync has option for deleting files which are on A and not in B.

The only thing is if you want to check which files are going to be deleted from B before getting deleted. For that can you check running dry run with --delete option and check what o/p comes?

One better approach is to setup backup policy as
  • Delete/Archive everything on B on Friday night, depending upon disk space.
  • Take full backup on Sat/sun.
  • Run incremental backup everyday till next Sat/Sun.
I know, you would have more queries, most welcome. Smilie
# 3  
Old 09-07-2011
Update on code

Ok so i took into account what you siad about rsync and managed something but now i want to finish what i started...after redefining what i need to do to myself i came to the following:

This is what i need processFolder() to do :P
Code:
#process folder(folder) {
#       for all the entreies in this folder
#               if current_thing = file
#                       if file existis on client
#                               if size is no same
#                                       flag error
#                               else
#                                       OK
#                       else
#                               flag for deletion
#               elif current_thing = dir
#                       process folder(current_thing)
#               elif
#                       error - not file or folder
#       fi
#}

And this is what i currently have....

Code:
#!/bin/sh

processFolder() {

  while ($BF_VAL="1") do
                if [ -f $BF_VAL ]
                 then
                        echo "file"
                else
                        echo "dir"

fi
}

echo "Initilising variables"

#CF = Client File
#BS = Backup Folder
#D = Dry-run

CF_ARG=0
CF_VAL=
BF_ARG=0
BF_VAL=
D_ARG=0

parseArgs()
{
while [ $# -gt 0 ]
do
    case "$1" in
                -d)   D_ARG=1;;
        -c)   CF_ARG=1; CF_VAL=$2; shift;;
        -b)   BF_ARG=1; BF_VAL=$2; shift;;
        ?) printHelp;;
        *) echo "$1 not an option"; printHelp;;
    esac
    shift
done
}

printHelp()
{
        echo "Usage: ./test-v2.sh -d -c [filename] -b [folder location]"
                echo ""
                echo "-d = Dry-run."
                echo "          ie. Run through the program as normal but only output a file with
                the changes that could be made."
                echo ""
                echo "-c = Client File input."
                echo "          ie. Supply the location and name of the previously generated client
                file list."
                echo ""
                echo "-b = Backup Folder Location."
                echo "          ie. Supply the location of where the backup is currently stored."
}

# main starts here

parseArgs $*

echo "Reading client file " $CF_VAL
echo "Reading backup folder " $BF_VAL
processFolder $BF_VAL

Any furthur input on this? ASAP but i know you all have lives Smilie

Regards,
John
# 4  
Old 09-21-2011
Updated code

Ok so i have made some progress but again im stumped... this is what i have so far:

Code:
#!/bin/sh
#Version 3.1
#Program to rationalise the off site backup in comparison to the current client file system.

processFolder() {
        if !(test -d "$1") then
                echo "do stuff with file"
                # Use cases for file checking:
                #       a) SRC has a file DEST doesn't < Warning report
                #       b) DEST has a file SRC doesn't < Mark for deletion
                #       c) SRC has a bigger size file to DEST < Warning report
                #       d) DEST has a bigger file size to SRC < Warning report
                #*      e) SRC has a diff date file to DEST < Warning report
                #       f) DEST has a directory SRC doesn't < Mark for deletion

                # Pesudo Code for file checking:
                #       if exits on SRC = yes then
                #               if same size = yes then
                #                       process_folder
                #               if same size = no then #c), d)
                #                       flag warning
                #       if exist on SRC = no then #b)
                #               mark for deletion
                #
                #elif item = directory then
                #       if exists on SRC = yes then
                #               process_folder
                #       if exists on SRC = no then #f)
                #               mark for deletion
                #
                #else
                #       echo "'current item' Is not file or directory"               


 
                # Existence checking
                # Need help here:
                # Does file exist on client server?

                # Size Checking               
                # BAKS contains size of file1
                # CLNTS contains size of file2
                BAKS=$(stat -c%s "$1")
                CLNTS=$(stat -c%s "$2")
                if [[ "$BAKS" -lt "$CLNTS" ]]; then
                          echo "Bbackup file is smaller"
                          echo "Warning, backup not successful"
                elif [[ "$BAKS" -gt "$CLNTS" ]]; then
                         echo "Client file size is smaller"
                         echo "Warning, backup not successful"
                elif [[ "$BAKS" -eq "$CLNTS" ]]; then
                         processFolder ()
                fi
               
        else
                cd $1
                for i in *
                do
                        processFolder "$i"
                done
        fi
}

parseArgs()
{
                if [  $# -lt 4 -o $# -gt 5 ]; then
                        printHelp
                        return 1;
                else

                        while [ $# -gt 0 ]
                        do
                                case "$1" in
                                        -d)   D_ARG=1;;
                                        -c)   CF_ARG=1; CF_VAL=$2; shift;;
                                        -b)   BF_ARG=1; BF_VAL=$2; shift;;
                                        ?) printHelp;;
                                        *) echo "$1 not an option"; printHelp;;
                                esac
                                shift
                        done
                fi
}

printHelp()
{
        echo "Usage: $0 -d -c [filename] -b [folder location]"
        echo ""
        echo "-d = Dry-run."
        echo "          ie. Run through the program as normal but only output a file with the changes that could be made."
        echo ""
        echo "-c = Client File input."
        echo "          ie. Supply the location and name of the previously generated client file list."
        echo ""
        echo "-b = Backup Folder Location."
        echo "          ie. Supply the location of where the backup is currently stored."

}

#main starts here

echo "Welcome to version 3.1"
echo "Initilising variables"
echo ""

#CF = Client File
#BS = Backup Folder
#D = Dry-run
CF_ARG=0
CF_VAL=
BF_ARG=0
BF_VAL=
D_ARG=0

parseArgs $*
PARGS=$?

if [ $PARGS -eq 0 ]; then

        echo "Reading client file " $CF_VAL
        echo "Reading backup folder " $BF_VAL
        processFolder $BF_VAL
        return $?

else
        return $PARGS
fi

So what im aiming to do is the commening at the top just inside of processFolder ()

Hope you guys can help here.....
# 5  
Old 09-27-2011
Bump

Still need help with this one
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help required to get a backup script working

Hi all I have a unix based firewall, which creates a daily backup file on the device. I need a script to scp this file over to a remote server. I can get this working daily using a basic script and a cron job. However, I only want it to send the latest config back up file and currently... (4 Replies)
Discussion started by: jimmyzoom
4 Replies

2. AIX

GTAR - new ways to faster backup - help required

We are taking backup of our application data(cobol file system, AIX/unix) before and after EOD job runs. The data size is approximately 260 GB in biggest branch. To reduce the backup time, 5 parallel execution is scheduled through control-m which backups up the files in 5 different *.gz. The job... (8 Replies)
Discussion started by: Bharath_79
8 Replies

3. AIX

GTAR - new ways for faster backup - help required

We are taking backup of our application data(cobol file system, AIX/unix) before and after EOD job runs. The data size is approximately 260 GB in biggest branch. To reduce the backup time, 5 parallel execution is scheduled through control-m which backups up the files in 5 different *.gz. The job... (2 Replies)
Discussion started by: Bharath_79
2 Replies

4. UNIX for Dummies Questions & Answers

Perl Script:how to find how many parameters are required to run the script

How to find how many parameters are required to run a Perl script? (1 Reply)
Discussion started by: Lakshman_Gupta
1 Replies

5. Shell Programming and Scripting

Help with Backup Shell Script for Network Device Configuration backup

HI all, im new to shell scripting. need your guidence for my script. i wrote one script and is attached here Im explaining the requirement of script. AIM: Shell script to run automatically as per scheduled and backup few network devices configurations. Script will contain a set of commands... (4 Replies)
Discussion started by: saichand1985
4 Replies

6. Shell Programming and Scripting

rsync backup mode(--backup) Are there any options to remove backup folders on successful deployment?

Hi Everyone, we are running rsync with --backup mode, Are there any rsync options to remove backup folders on successful deployment? Thanks in adv. (0 Replies)
Discussion started by: MVEERA
0 Replies

7. Shell Programming and Scripting

Help required to get a script

Hi Experts, I am very beginner in Bash Shell Scripting. Can anyone please guide me to create a script which should show the most busy file systems in sort basis as there are a lot of file systems on the server. I was told this task to be done by my IT lead and I must have to do this in... (3 Replies)
Discussion started by: naw_deepak
3 Replies

8. Shell Programming and Scripting

Getting required fields from a test file in required fromat in unix

My data is something like shown below. date1 date2 aaa bbbb ccccc date3 date4 dddd eeeeeee ffffffffff ggggg hh I want the output like this date1date2 aaa eeeeee I serached in the forum but didn't find the exact matching solution. Please help. (7 Replies)
Discussion started by: rdhanek
7 Replies

9. Shell Programming and Scripting

script required

The line is like this +abc+def+mgh+ddsdsd+sa i.e. words seperated by +. There is a plus in the beginning. i want to conver this line to abc, def, mgh, ddsdsd, sa please provide the logic in the form of a shell script Thanks in advance (13 Replies)
Discussion started by: skyineyes
13 Replies

10. Shell Programming and Scripting

Script help required!

Hi there, i am trying to create a script that checks for the existence of users on the system, if they exist then their details should print on the screen with a message that the id is in use. I am having a bit of trouble with it. Any ideas? Cheers Kev! (8 Replies)
Discussion started by: kev112
8 Replies
Login or Register to Ask a Question