rsync: taking advantage of files in different directory other than destination


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users rsync: taking advantage of files in different directory other than destination
# 1  
Old 05-05-2008
rsync: taking advantage of files in different directory other than destination

Dear Folks,

I have to backup pgsql database dump everynight on a routine. The database dump actually contains sql(text) statements. The actual size of the database dump is aroung 800 MB. Between two days backup, only few lines of statements are modified/added/deleted.

I dont want to do incremental backup. Instead, I want everyday's full dump to be stored on to a folder named DD-MM-YY(ex: 01-05-08). Everyday there will be a folder created with name DD-MM-YY and the dump will be stored there.

Since the difference between the two day's backup are pretty minimal, I dont want to resend the 800 MB files everyday.

Suppose,

1. I have taken yesterdays dump in a folder named 04-05-08 in database server
2. I have rsynced the folder 04-05-08 to the backup server.
3. I have taken today's dump in folder named 05-05-08 in database server.
4. Now while doing rsync, I want rsync to compare the <database server's 05-05-08/dump file> with the <04-05-08/dump file> which already exists in backup server. (then eventually, I want rsync to send only the differences and store the output in folder 05-05-08).

Typicaly I want to take advantage of yesterday's file which already present in a different folder.

Can anyone suggest a working method to implement this ? (I tried --copy-dest / --compare-dest .. but couldnt get it to work)

Any response in this regard is most appreciated.

Regards,
rssrik
# 2  
Old 05-06-2008
Rsync - with a twist of lemon...

Let's low tech this for a minute and test this by doing the steps manually. I am *assuming* the dump file name is the same everyday, or it has simple enough naming convention that you can make a script to rename a file.

1) You already have rsync'd and you have a folder named 04-05-08 on the backup server. It's now time to get the 05-05-08 version, and magically you know the DB Server is done creating today's dumpfile....
2) On the backup server, cp -rp ./04-05-08 ./05-05-08
-- you now have a new folder to rsync into, and it has yesterday's version of the dump file itself, with yesterday's name and we've preserved the timestamps and permissions on the files.
3) If necessary, rename the ./05-05-08/dumpfile to match today's dump name.
3) start your rsync between the 05-05-08 folders.

To quote the rsync man page:
If any of the files already exist on the remote system then the rsync remote-update protocol is used to update the file by sending only the differences. Your file already exists, so it will run update instead of a copy.

With any luck, the compare will take less time than an actual copy normally does. No matter what, 800MB is a lot to accomplish.


Important Question: Does your pgsql dump actually need to create a NEW 800MB file every day on the DB server, or can it just update the existing dumpfile?

If you can update the same "dump" instance, then I'd just start an rsync of the master dumpfile, and on the backup server just schedule a cp of the the currently rsync'd instance over to the daily folder. Schedule the rsync for every 3 hours, and you have a decent current backup, plus yesterday's copy on the shelf.

Please remember: If you are out of disk space, or your dumpfile failed to be created, it's really hard to finish a backup, so script with that in mind, or confirm something is monitoring your diskspace....

A backup is a terrible thing to lose.Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Rsync - delete extra files in Destination without synchronising directories

I have a script that synchronises a directory to a DR server, but to improve the time, I actually use rsync to transfer files * in one batch and also * in another batch - both batches run from the same script and run in the background. My problem is that there isn't much space on the... (1 Reply)
Discussion started by: Catullus
1 Replies

2. UNIX for Dummies Questions & Answers

How to move gz files from one source directory to destination directory?

Hi All, Daily i am doing the house keeping in one of my server and manually moving the files which were older than 90 days and moving to destination folder. using the find command . Could you please assist me how to put the automation using the shell script . ... (11 Replies)
Discussion started by: venkat918
11 Replies

3. UNIX for Dummies Questions & Answers

Rsync not deleting destination files/folders

Hi All, I want delete all files from destination folder. I am trying below commands rsync -avFF --delete-after /home/vizion/source/ /home/vizion/destination/ I need to be delete all file/fodders from destination Any body have idea ? Please correct me. (3 Replies)
Discussion started by: Chenchireddy
3 Replies

4. UNIX for Advanced & Expert Users

Rsync error while running from destination to source

hi All, i have 2 server setup now for Rsync, i configured Rsync on both of the server and it worked well when i did run from source to destination. and while running back from destination to source it produced this error: bash-3.2$ ksh rsync_bravo_db.ksh usa0300uz1247.apps.mc.xerox.com... (0 Replies)
Discussion started by: lovelysethii
0 Replies

5. Shell Programming and Scripting

Rsync not deleting files on destination

I've got a rsync script that backups file from disk1 to disk3. This works great however if there are extra files on disk3 they do not get deleted by the sync. I'm syncing all folders from F-J rsync --progress -v --delete --delete-excluded -av --recursive /home/disk1/Files/*... (0 Replies)
Discussion started by: gmccarthy
0 Replies

6. Shell Programming and Scripting

Move all files from source to destination directory based on the filename

Move all files starting with a specific name to different directory. This shell script program should have three parameters File Name Source Directory Destination Directory User should be able to enter ‘AB_CD*' in file name parameter. In this case all the files starting with AB_CD will... (1 Reply)
Discussion started by: chetancrsp18
1 Replies

7. UNIX for Advanced & Expert Users

How to rsync or tar directory trees, with hidden directory, but without files?

I want to backup all the directory tress, including hidden directories, without copying any files. find . -type d gives the perfect list. When I tried tar, it won't work for me because it tars all the files. find . -type d | xargs tar -cvf a.tar So i tried rsync. On my own test box, the... (4 Replies)
Discussion started by: fld2007
4 Replies

8. Shell Programming and Scripting

recursive searching for files in directory that matches a particular name - taking care of links

Hi, I am writing a shell script that finds all files named <myFile> in a directory <dir> or any of its subdirectories, recursively. I also need to take care of symbolic links that may form cycles, to avoid infinite loops. I started writing the code but got stuck. I thought using recursion... (7 Replies)
Discussion started by: vickylife
7 Replies

9. Shell Programming and Scripting

[BASH] rsync - error on destination

Hi everyone, and thanks to all for your assistance. I have a problem with the rsync command. I want to make a backup of a "source" directory in a "destination" directory. I want to specify: "absolute path of destination" (identified by a ~): ~/Destination or a "relative path of... (0 Replies)
Discussion started by: PaganoM
0 Replies
Login or Register to Ask a Question