02-28-2016
Well the only way to speed up the process is not to copy entire hierarchy every time a script is ran.
Have you considered using
rsync , since the data is static it will only copy changes every time it is ran ?
Using rsync the script will probably be couple of lines long.
How big is the directory (size and number of files) and how much bandwidth do you have ?
Perhaps you could also speed up the process with various mount options for NFS or setting atime off for the source mountpoint, but one can only guess.
Other approach is to have a checksum table of all files (source and destination) in perhaps sqlite3 database or a plain text file which you will ran you script against so it only copies the changes.
This will require a bit more effort tho unlike couple of rsync lines
Hope that helps
Regards
Peasant.
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I have been tasked with getting an AIX 4.3.3 box to backup to a NAS applicance device which provides NFS service. It is an intermediary repository so that other tools can transport the resulting backup file to another NAS Applicance at a remote site on a secondary frame connection.
Anyone have... (10 Replies)
Discussion started by: sirhisss
10 Replies
2. UNIX for Advanced & Expert Users
Dear All
I am using cat command for concatenating multiple files. Some time i also use append command when there are few files.
Is there faster way of concatenating multiple files(60 to 70 files) each of
156 MB or less/more.:)
Thanx (1 Reply)
Discussion started by: tkbharani
1 Replies
3. UNIX for Dummies Questions & Answers
Hi guys,
I have a redhat laptop and a sun solaris 8 server networked together
I created an nfs share on the sun server and backed up an image of the Redhat laptop to it.
The Hard disk size of the laptop is 40Gb but I have about 38Gb free space on the sun server. So I compressed the image... (9 Replies)
Discussion started by: Stin
9 Replies
4. Shell Programming and Scripting
Hi All,
I have some 80,000 files in a directory which I need to rename. Below is the command which I am currently running and it seems, it is taking fore ever to run this command. This command seems too slow. Is there any way to speed up the command. I have have GNU Parallel installed on my... (6 Replies)
Discussion started by: shoaibjameel123
6 Replies
5. Shell Programming and Scripting
Hi Everyone,
we are running rsync with --backup mode, Are there any rsync options to remove backup folders on successful deployment?
Thanks in adv. (0 Replies)
Discussion started by: MVEERA
0 Replies
6. Shell Programming and Scripting
Hello,
I was wondering if anyone knows a faster way to search and compare strings and dates from 2 files?
I'm currently using "for loop" but seems sluggish as i have to cycle through 10 directories with 10 files each containing thousands of lines.
Given:
-10 directories
-10 files... (4 Replies)
Discussion started by: agentgrecko
4 Replies
7. Shell Programming and Scripting
Good evening
Im new at unix shell scripting and im planning to script a shell that removes headers for about 120 files in a directory and each file contains about 200000
lines in average.
i know i will loop files to process each one and ive found in this great forum different solutions... (5 Replies)
Discussion started by: alexcol
5 Replies
8. AIX
We are taking backup of our application data(cobol file system, AIX/unix) before and after EOD job runs. The data size is approximately 260 GB in biggest branch. To reduce the backup time, 5 parallel execution is scheduled through control-m which backups up the files in 5 different *.gz. The job... (2 Replies)
Discussion started by: Bharath_79
2 Replies
9. AIX
We are taking backup of our application data(cobol file system, AIX/unix) before and after EOD job runs. The data size is approximately 260 GB in biggest branch. To reduce the backup time, 5 parallel execution is scheduled through control-m which backups up the files in 5 different *.gz. The job... (8 Replies)
Discussion started by: Bharath_79
8 Replies
10. UNIX for Advanced & Expert Users
I have a very big input file <inputFile1.txt> which has list of mobile no
inputFile1.txt
3434343
3434323
0970978
85233
... around 1 million records
i have another file as inputFile2.txt which has some log detail big file
inputFile2.txt
afjhjdhfkjdhfkd df h8983 3434343 | 3483 | myout1 |... (3 Replies)
Discussion started by: reldb
3 Replies
LEARN ABOUT DEBIAN
svn::notify::mirror::rsync
SVN::Notify::Mirror::Rsync(3pm) User Contributed Perl Documentation SVN::Notify::Mirror::Rsync(3pm)
NAME
SVN::Notify::Mirror::Rsync - Mirror a repository path via Rsync
SYNOPSIS
Use svnnotify in post-commit:
svnnotify --repos-path "$1" --revision "$2"
--handler Mirror::Rsync --to "/path/to/local/htdocs"
[--svn-binary /full/path/to/svn]
--rsync-host remote_server
[--rsync-delete=[yes|no]]
[--rsync-dest "/path/on/remote/server"]
[--rsync-args arg1 [--rsync-args arg2...]]
[[--rsync-ssh] [--ssh-user remote_user]
[--ssh-identity /home/user/.ssh/id_rsa]]
or better yet, use SVN::Notify::Config for a more sophisticated setup:
#!/usr/bin/perl -MSVN::Notify::Config=$0
--- #YAML:1.0
'':
PATH: "/usr/bin:/usr/local/bin"
'path/in/repository':
handler: Mirror
to: "/path/to/www/htdocs"
'some/other/path/in/repository':
handler: Mirror
to: "/path/to/local/www/htdocs"
rsync-host: "remote_host"
rsync-dest: "/path/on/remote/www/htdocs"
ssh-user: "remote_user"
ssh-identity: "/home/user/.ssh/id_rsa"
DESCRIPTION
Keep a directory in sync with a portion of a Subversion repository. Typically used to keep a development web server in sync with the
changes made to the repository. This directory can either be on the same box as the repository itself, or it can be remote (via SSH
connection).
USAGE
Depending on whether the target is a "Local Mirror" or a Remote Mirror, there are different options available. All options are available
either as a commandline option to svnnotify or as a hash key in SVN::Notify::Config (see their respective documentation for more details).
Working Copy on Local host
Because 'svn export' is not able to be consistently updated, the local rsync'd directory must be a full working copy. The remote server
will only contain the ordinary files (no Subversion admin files).
The files in the working copy must be writeable (preferrably owned) by the user identity executing the hook script (this is the user
identity that is running Apache or svnserve respectively).
Local Mirror
Please see " SVN::Notify::Mirror " for details.
Remote Mirror
Used for directories not located on the same machine as the repository itself. Typically, this might be a production web server located in
a DMZ, so special consideration must be paid to security concerns. In particular, the remote mirror server may not be able to directly
access the repository box.
o rsync-host
This value is required and must be the hostname or IP address of the remote host (where the mirror directories reside).
o rsync-delete
The default mode of operation is to delete remote files which are not present in the local working copy. NOTE: this will delete any
unversioned files in the remote directory tree. Unless you have all of your files under version control, you should pass the
"--no-rsync-delete" or "--rsync-delete no" option.
o rsync-dest
This optional value specifies the path to update on the remote host. If you do not specify this value, the same path as passed in as
the "--to" parameter will be used (this may not be what you meant to do).
o rsync-args
This optional parameter can be used to pass additional commandline options to the rsync command. You can use this multiple times in
order to pass multiple options. The default args are "--archive --compress". See the "rsync-ssh" options for using SSH instead of RSH
(rather than pass those commands via "--rsync-args"
o rsync-ssh
This optional parameter signals that you wish to use SSH instead of whatever the default remote shell program is configured in your
copy of rsync. You may need to set one or more of the "ssh-*" parameters as well.
o ssh-user
If the remote user is different than the local user executing the postcommit script, you can specify it with this parameter. You would
often use this in conjunction with the next parameter.
o ssh-identity
This value may be optional and should be the full path to the local identity file being used to authenticate with the remote host. If
you are setting the ssh-user to be something other than the local user name, you will typically also have to set the ssh-identity.
AUTHOR
John Peacock <jpeacock@cpan.org>
COPYRIGHT
Copyright (c) 2005-2008 John Peacock
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
The full text of the license can be found in the LICENSE file included with this module.
SEE ALSO
SVN::Notify, SVN::Notify::Config, SVN::Notify::Mirror
perl v5.14.2 2012-07-04 SVN::Notify::Mirror::Rsync(3pm)