Sponsored Content
Top Forums UNIX for Advanced & Expert Users rsync: taking advantage of files in different directory other than destination Post 302192166 by wildkaarde on Tuesday 6th of May 2008 07:35:36 AM
Old 05-06-2008
Rsync - with a twist of lemon...

Let's low tech this for a minute and test this by doing the steps manually. I am *assuming* the dump file name is the same everyday, or it has simple enough naming convention that you can make a script to rename a file.

1) You already have rsync'd and you have a folder named 04-05-08 on the backup server. It's now time to get the 05-05-08 version, and magically you know the DB Server is done creating today's dumpfile....
2) On the backup server, cp -rp ./04-05-08 ./05-05-08
-- you now have a new folder to rsync into, and it has yesterday's version of the dump file itself, with yesterday's name and we've preserved the timestamps and permissions on the files.
3) If necessary, rename the ./05-05-08/dumpfile to match today's dump name.
3) start your rsync between the 05-05-08 folders.

To quote the rsync man page:
If any of the files already exist on the remote system then the rsync remote-update protocol is used to update the file by sending only the differences. Your file already exists, so it will run update instead of a copy.

With any luck, the compare will take less time than an actual copy normally does. No matter what, 800MB is a lot to accomplish.


Important Question: Does your pgsql dump actually need to create a NEW 800MB file every day on the DB server, or can it just update the existing dumpfile?

If you can update the same "dump" instance, then I'd just start an rsync of the master dumpfile, and on the backup server just schedule a cp of the the currently rsync'd instance over to the daily folder. Schedule the rsync for every 3 hours, and you have a decent current backup, plus yesterday's copy on the shelf.

Please remember: If you are out of disk space, or your dumpfile failed to be created, it's really hard to finish a backup, so script with that in mind, or confirm something is monitoring your diskspace....

A backup is a terrible thing to lose.Smilie
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

[BASH] rsync - error on destination

Hi everyone, and thanks to all for your assistance. I have a problem with the rsync command. I want to make a backup of a "source" directory in a "destination" directory. I want to specify: "absolute path of destination" (identified by a ~): ~/Destination or a "relative path of... (0 Replies)
Discussion started by: PaganoM
0 Replies

2. Shell Programming and Scripting

recursive searching for files in directory that matches a particular name - taking care of links

Hi, I am writing a shell script that finds all files named <myFile> in a directory <dir> or any of its subdirectories, recursively. I also need to take care of symbolic links that may form cycles, to avoid infinite loops. I started writing the code but got stuck. I thought using recursion... (7 Replies)
Discussion started by: vickylife
7 Replies

3. UNIX for Advanced & Expert Users

How to rsync or tar directory trees, with hidden directory, but without files?

I want to backup all the directory tress, including hidden directories, without copying any files. find . -type d gives the perfect list. When I tried tar, it won't work for me because it tars all the files. find . -type d | xargs tar -cvf a.tar So i tried rsync. On my own test box, the... (4 Replies)
Discussion started by: fld2007
4 Replies

4. Shell Programming and Scripting

Move all files from source to destination directory based on the filename

Move all files starting with a specific name to different directory. This shell script program should have three parameters File Name Source Directory Destination Directory User should be able to enter ‘AB_CD*' in file name parameter. In this case all the files starting with AB_CD will... (1 Reply)
Discussion started by: chetancrsp18
1 Replies

5. Shell Programming and Scripting

Rsync not deleting files on destination

I've got a rsync script that backups file from disk1 to disk3. This works great however if there are extra files on disk3 they do not get deleted by the sync. I'm syncing all folders from F-J rsync --progress -v --delete --delete-excluded -av --recursive /home/disk1/Files/*... (0 Replies)
Discussion started by: gmccarthy
0 Replies

6. UNIX for Advanced & Expert Users

Rsync error while running from destination to source

hi All, i have 2 server setup now for Rsync, i configured Rsync on both of the server and it worked well when i did run from source to destination. and while running back from destination to source it produced this error: bash-3.2$ ksh rsync_bravo_db.ksh usa0300uz1247.apps.mc.xerox.com... (0 Replies)
Discussion started by: lovelysethii
0 Replies

7. UNIX for Dummies Questions & Answers

Rsync not deleting destination files/folders

Hi All, I want delete all files from destination folder. I am trying below commands rsync -avFF --delete-after /home/vizion/source/ /home/vizion/destination/ I need to be delete all file/fodders from destination Any body have idea ? Please correct me. (3 Replies)
Discussion started by: Chenchireddy
3 Replies

8. UNIX for Dummies Questions & Answers

How to move gz files from one source directory to destination directory?

Hi All, Daily i am doing the house keeping in one of my server and manually moving the files which were older than 90 days and moving to destination folder. using the find command . Could you please assist me how to put the automation using the shell script . ... (11 Replies)
Discussion started by: venkat918
11 Replies

9. UNIX for Advanced & Expert Users

Rsync - delete extra files in Destination without synchronising directories

I have a script that synchronises a directory to a DR server, but to improve the time, I actually use rsync to transfer files * in one batch and also * in another batch - both batches run from the same script and run in the background. My problem is that there isn't much space on the... (1 Reply)
Discussion started by: Catullus
1 Replies
SVN::Notify::Mirror::Rsync(3pm) 			User Contributed Perl Documentation			   SVN::Notify::Mirror::Rsync(3pm)

NAME
SVN::Notify::Mirror::Rsync - Mirror a repository path via Rsync SYNOPSIS
Use svnnotify in post-commit: svnnotify --repos-path "$1" --revision "$2" --handler Mirror::Rsync --to "/path/to/local/htdocs" [--svn-binary /full/path/to/svn] --rsync-host remote_server [--rsync-delete=[yes|no]] [--rsync-dest "/path/on/remote/server"] [--rsync-args arg1 [--rsync-args arg2...]] [[--rsync-ssh] [--ssh-user remote_user] [--ssh-identity /home/user/.ssh/id_rsa]] or better yet, use SVN::Notify::Config for a more sophisticated setup: #!/usr/bin/perl -MSVN::Notify::Config=$0 --- #YAML:1.0 '': PATH: "/usr/bin:/usr/local/bin" 'path/in/repository': handler: Mirror to: "/path/to/www/htdocs" 'some/other/path/in/repository': handler: Mirror to: "/path/to/local/www/htdocs" rsync-host: "remote_host" rsync-dest: "/path/on/remote/www/htdocs" ssh-user: "remote_user" ssh-identity: "/home/user/.ssh/id_rsa" DESCRIPTION
Keep a directory in sync with a portion of a Subversion repository. Typically used to keep a development web server in sync with the changes made to the repository. This directory can either be on the same box as the repository itself, or it can be remote (via SSH connection). USAGE
Depending on whether the target is a "Local Mirror" or a Remote Mirror, there are different options available. All options are available either as a commandline option to svnnotify or as a hash key in SVN::Notify::Config (see their respective documentation for more details). Working Copy on Local host Because 'svn export' is not able to be consistently updated, the local rsync'd directory must be a full working copy. The remote server will only contain the ordinary files (no Subversion admin files). The files in the working copy must be writeable (preferrably owned) by the user identity executing the hook script (this is the user identity that is running Apache or svnserve respectively). Local Mirror Please see " SVN::Notify::Mirror " for details. Remote Mirror Used for directories not located on the same machine as the repository itself. Typically, this might be a production web server located in a DMZ, so special consideration must be paid to security concerns. In particular, the remote mirror server may not be able to directly access the repository box. o rsync-host This value is required and must be the hostname or IP address of the remote host (where the mirror directories reside). o rsync-delete The default mode of operation is to delete remote files which are not present in the local working copy. NOTE: this will delete any unversioned files in the remote directory tree. Unless you have all of your files under version control, you should pass the "--no-rsync-delete" or "--rsync-delete no" option. o rsync-dest This optional value specifies the path to update on the remote host. If you do not specify this value, the same path as passed in as the "--to" parameter will be used (this may not be what you meant to do). o rsync-args This optional parameter can be used to pass additional commandline options to the rsync command. You can use this multiple times in order to pass multiple options. The default args are "--archive --compress". See the "rsync-ssh" options for using SSH instead of RSH (rather than pass those commands via "--rsync-args" o rsync-ssh This optional parameter signals that you wish to use SSH instead of whatever the default remote shell program is configured in your copy of rsync. You may need to set one or more of the "ssh-*" parameters as well. o ssh-user If the remote user is different than the local user executing the postcommit script, you can specify it with this parameter. You would often use this in conjunction with the next parameter. o ssh-identity This value may be optional and should be the full path to the local identity file being used to authenticate with the remote host. If you are setting the ssh-user to be something other than the local user name, you will typically also have to set the ssh-identity. AUTHOR
John Peacock <jpeacock@cpan.org> COPYRIGHT
Copyright (c) 2005-2008 John Peacock This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. The full text of the license can be found in the LICENSE file included with this module. SEE ALSO
SVN::Notify, SVN::Notify::Config, SVN::Notify::Mirror perl v5.14.2 2012-07-04 SVN::Notify::Mirror::Rsync(3pm)
All times are GMT -4. The time now is 03:01 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy