Compare & Copy Directories : Bash Script Help


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Compare & Copy Directories : Bash Script Help
# 1  
Old 12-14-2012
Compare & Copy Directories : Bash Script Help

Beginner/Intermediate shell; comfortable in the command line.

I have been looking for a solution to a backup problem. I need to compare Directory 1 to Directory 2 and copy all modified or new files/directories from Directory 1 to Directory 3. I need the directory and file structure to be mirrored on Directory 3. Another way of thinking about the logic is: Dir1 - Dir2 = Dir3. I want Dir3 to be portable between users/machines so I don't think an incremental backup with hardlinks will work. Seems like it should be easy right?

I found a scripts that seems like it could do the trick. However, I'm having some trouble getting it to work properly. It compares Dir1 to Dir2 and copies all new or modified files into Dir3 but it does NOT recreate the directory structure on the target. Instead all files end up in a flat folder.

From Statckoverflow . com

Code:
#!/bin/bash
#
# setup folders for our different stages
#
Dir1=/Users/username/source1
Dir2=/Users/username/source2
Dir3=/Users/username/target
#
cd $Dir1
for f in *f
do
# Diff the files - ignore the output...
    diff $f $Dir2 > /dev/null 2>&1
# ...but get the status
    status=$?
    if [ $status -eq 0 ] ; then
# Files are identical - don't copy the file
        echo $f unchanged
    elif [ $status -eq 1 ] ; then
# Files differ - copy new file
        echo $f changed
        cp $f $diff
    elif [ $status -eq 2 ] ; then
# Old file doesn't exist - copy new file
        echo old $f does not exist
        cp $f $diff
    fi
done

I started my reading with rsync. I've used it for back up - locally and over ssh between NAS servers. It should be able to do the job, right? I just couldn't find examples to guide me and I don't know enough and / or have enough time to do it unaided. Researching it I got side tracked by diff which I didn't know much about but it looked promising.

My preference is to do this with a bash script. Anyone have any ideas?

Thanks!

Last edited by Rod; 12-14-2012 at 09:56 PM..
# 2  
Old 12-14-2012
What OS & do you have the stat command?
# 3  
Old 12-14-2012
I'm working in OpenBSD (OSX) which does support 'stat'.
# 4  
Old 12-15-2012
Rod, here is a crude script that I wrote in bash. It uses cksum to check if the file in DIR_1 is modified. I have commented most of the steps. This one is just to give you an idea of using a different approach other than diff, I hope it helps.
Code:
#!/bin/bash

for file_1 in $( find DIR_1/* -type f | sed 's/\.\.\///g' )             # For each file in DIR_1
do
        file_2=$( echo $file_1 | sed 's/DIR_1/DIR_2/g' )                # Getting file path in DIR_2
        dir_2=$( dirname $file_2 )

        if [ ! -d $dir_2 ]                                              # Checking if sub-dir exists in DIR_2
        then
                echo -e "Dir: $dir_2 does not exist. Creating...\c"
                mkdir -p $dir_2                                         # Creating if sub-dir missing
                echo "Done"
        fi

        if [ -f $file_2 ]                                               # Checking if file exists in DIR_2
        then
                cksum_file_1=$( cksum $file_1 | cut -f 1 -d " " )       # Get cksum of file in DIR_1
                cksum_file_2=$( cksum $file_2 | cut -f 1 -d " " )       # Get cksum of file in DIR_2

                if [ $cksum_file_1 -ne $cksum_file_2 ]                  # Check if cksum matches
                then
                        echo -e "File: $file_1 is modified. Copying..\c"
                        cp $file_1 $file_2                              # Copy if cksum mismatch
                        echo "Done"
                fi
        else
                echo -e "File: $file_2 does not exist. Copying...\c"
                cp $file_1 $file_2                                      # Copy if file does not exist.
                echo "Done"
        fi
done

BTW I am using only 2 directory structures in this script: DIR_1 & DIR_2.

Last edited by Yoda; 12-15-2012 at 01:31 AM..
# 5  
Old 12-15-2012
Compare & Copy Directories : Bash Script Help

Hi Rod

You can make a script in order to get all the differences between Dir1 and Dir2 with the rsync parameter --dry-run, after that, you only need to do a copy of this files/folders from Dir1 to Dir3, something like that in csh:

Code:
#!/bin/csh
setenv Dir1 "path1"
setenv Dir2 "path2"
setenv Dir3 "path3"
rsync -Havx --dry-run "$Dir1" "$Dir2" > /tmp/differences
foreach "element" ( 'cat /tmp/differences' )
            cp -rpf "$element" "$Dir3"
end
exit

This script, or something like that, will copy any file or folder from the Dir1 to Dir3 with the same structure.

BR

Hombreopaco

Last edited by hombreopaco; 12-15-2012 at 08:23 AM.. Reason: code tags
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Compare directories and copy differences (files) in a another directory

Hey im working on script that can compare 2 directory and check difference, then copy difference files in third diretory. here is the story: in folder one we have 12 subfolder and in each of them near 500 images hosted. 01 02 03 04 05 06 07 08 09 10 11 12 in folder 2 we have same subfolder... (2 Replies)
Discussion started by: nimafire
2 Replies

2. Shell Programming and Scripting

Need shell script to compare directories and delete files on target server

Hello, I need help in writing the shell script for below mentioned case. There are 2 servers(server A, server B). A cronjob syncs files between these 2 servers. Existing script is copying files from A to B. This is done using the command rsync. However, the files are not deleted... (2 Replies)
Discussion started by: SravaniVedam11
2 Replies

3. Shell Programming and Scripting

Shell script to copy particular file from directories recursively

I have directory path in which there are several sub directories. In all these sub dir there will be one env.cnf file. I want to copy this env.cnf file from each sub dir's and place them in destination path by creating same filename as sub dir_env.cnf. After copying env.cnf files from source... (4 Replies)
Discussion started by: Optimus81
4 Replies

4. Shell Programming and Scripting

Compare File & Copy Replace if Successful

Hi All, I have written a shell script that creates a backup of my MySQL database. The script performs the following functions: Creates a Backup of the MySQL database Compresses the Backup Copies the Backup to a Remote Server Send an E-Mail displaying the size of the Backup Removes any... (6 Replies)
Discussion started by: SalientAnimal
6 Replies

5. Shell Programming and Scripting

Script to copy certain info from several directories

Hi, I am writing a script to copy certain file name in txt file . It is working fine if I provide a single directory name (for example "/eos/uscms/store/user/pooja04//analysis2012/525/data/doubleele/2012/datav1/" ) where those specific files are present ending with *root . But I want to modify... (14 Replies)
Discussion started by: nrjrasaxena
14 Replies

6. Shell Programming and Scripting

Need a shell script to compare two directories and produces the output

Hi, I am using solaris OS 10 and Bash shell.I need a script which will compare the two directories and produces the output. Step 1: In detail say suppoose I have machine one and have a directory dir1. Script should iterate through the directories and subdirectories inside and produce the output... (10 Replies)
Discussion started by: muraliinfy04
10 Replies

7. UNIX for Dummies Questions & Answers

Compare two directories and copy differing files

Hello there, I'm a total noob to shell scripting. :) What I want to do is compare the contents of Folder A and Folder B, and copy any files in Folder A that do not exist in Folder B over to Folder B. I have gotten so far as: diff -rq folderA folderB which returns the names of the files,... (3 Replies)
Discussion started by: raaaaaa
3 Replies

8. Shell Programming and Scripting

Bash scripting to compare N number of files located in two directories

I want to compare "N" (around 2000+) number of huge files located in a directory A against "N" files located in a different directory using Bash scripting. Please help me with any scripts available. Thanks. (2 Replies)
Discussion started by: Sangtha
2 Replies

9. Shell Programming and Scripting

Bash Script to Read & Write on different directories

Hi, root@server] df -h 121G 14G 101G 12% /home 147G 126G 14G 91% /backup We having our site files and images are storing in /backup/home/user/files/ through symbolic link created in /home directory pointing in /backup directory as following. root@server] cd /home... (1 Reply)
Discussion started by: mirfan
1 Replies

10. Shell Programming and Scripting

How do i collect Date & Time from Different Directories in a script

How do i collect Date & Time from Different Directories in a script The script iam using for a single directory is : ls -l | grep awk '{print $8}' (2 Replies)
Discussion started by: laknar
2 Replies
Login or Register to Ask a Question