|
|
Sponsored Content
Operating Systems
SCO
Backup of files using NFS a faster way
Post 302967755 by Peasant on Sunday 28th of February 2016 02:30:47 AM
02-28-2016
Moderator
Well the only way to speed up the process is not to copy entire hierarchy every time a script is ran.
Have you considered using rsync , since the data is static it will only copy changes every time it is ran ?
Using rsync the script will probably be couple of lines long.
How big is the directory (size and number of files) and how much bandwidth do you have ?
Perhaps you could also speed up the process with various mount options for NFS or setting atime off for the source mountpoint, but one can only guess.
Other approach is to have a checksum table of all files (source and destination) in perhaps sqlite3 database or a plain text file which you will ran you script against so it only copies the changes.
This will require a bit more effort tho unlike couple of rsync lines
Hope that helps
Regards
Peasant.
Have you considered using rsync , since the data is static it will only copy changes every time it is ran ?
Using rsync the script will probably be couple of lines long.
How big is the directory (size and number of files) and how much bandwidth do you have ?
Perhaps you could also speed up the process with various mount options for NFS or setting atime off for the source mountpoint, but one can only guess.
Other approach is to have a checksum table of all files (source and destination) in perhaps sqlite3 database or a plain text file which you will ran you script against so it only copies the changes.
This will require a bit more effort tho unlike couple of rsync lines
Hope that helps
Regards
Peasant.
|
|
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I have been tasked with getting an AIX 4.3.3 box to backup to a NAS applicance device which provides NFS service. It is an intermediary repository so that other tools can transport the resulting backup file to another NAS Applicance at a remote site on a secondary frame connection.
Anyone have... (10 Replies)
Discussion started by: sirhisss
10 Replies
2. UNIX for Advanced & Expert Users
Dear All
I am using cat command for concatenating multiple files. Some time i also use append command when there are few files.
Is there faster way of concatenating multiple files(60 to 70 files) each of
156 MB or less/more.:)
Thanx (1 Reply)
Discussion started by: tkbharani
1 Replies
3. UNIX for Dummies Questions & Answers
Hi guys,
I have a redhat laptop and a sun solaris 8 server networked together
I created an nfs share on the sun server and backed up an image of the Redhat laptop to it.
The Hard disk size of the laptop is 40Gb but I have about 38Gb free space on the sun server. So I compressed the image... (9 Replies)
Discussion started by: Stin
9 Replies
4. Shell Programming and Scripting
Hi All,
I have some 80,000 files in a directory which I need to rename. Below is the command which I am currently running and it seems, it is taking fore ever to run this command. This command seems too slow. Is there any way to speed up the command. I have have GNU Parallel installed on my... (6 Replies)
Discussion started by: shoaibjameel123
6 Replies
5. Shell Programming and Scripting
rsync backup mode(--backup) Are there any options to remove backup folders on successful deployment?
Hi Everyone,
we are running rsync with --backup mode, Are there any rsync options to remove backup folders on successful deployment?
Thanks in adv. (0 Replies)
Discussion started by: MVEERA
0 Replies
6. Shell Programming and Scripting
Hello,
I was wondering if anyone knows a faster way to search and compare strings and dates from 2 files?
I'm currently using "for loop" but seems sluggish as i have to cycle through 10 directories with 10 files each containing thousands of lines.
Given:
-10 directories
-10 files... (4 Replies)
Discussion started by: agentgrecko
4 Replies
7. Shell Programming and Scripting
Good evening
Im new at unix shell scripting and im planning to script a shell that removes headers for about 120 files in a directory and each file contains about 200000
lines in average.
i know i will loop files to process each one and ive found in this great forum different solutions... (5 Replies)
Discussion started by: alexcol
5 Replies
8. AIX
We are taking backup of our application data(cobol file system, AIX/unix) before and after EOD job runs. The data size is approximately 260 GB in biggest branch. To reduce the backup time, 5 parallel execution is scheduled through control-m which backups up the files in 5 different *.gz. The job... (2 Replies)
Discussion started by: Bharath_79
2 Replies
9. AIX
We are taking backup of our application data(cobol file system, AIX/unix) before and after EOD job runs. The data size is approximately 260 GB in biggest branch. To reduce the backup time, 5 parallel execution is scheduled through control-m which backups up the files in 5 different *.gz. The job... (8 Replies)
Discussion started by: Bharath_79
8 Replies
10. UNIX for Advanced & Expert Users
I have a very big input file <inputFile1.txt> which has list of mobile no
inputFile1.txt
3434343
3434323
0970978
85233
... around 1 million records
i have another file as inputFile2.txt which has some log detail big file
inputFile2.txt
afjhjdhfkjdhfkd df h8983 3434343 | 3483 | myout1 |... (3 Replies)
Discussion started by: reldb
3 Replies
LEARN ABOUT DEBIAN
parallel-rsync
PARALLEL-RSYNC(1) PARALLEL-RSYNC(1) NAME
parallel-rsync - deploy files to listed hosts SYNOPSIS
parallel-rsync [OPTIONS] -h hosts.txt local remote DESCRIPTION
pssh provides a number of commands for executing against a group of computers, using SSH. It's most useful for operating on clusters of homogenously-configured hosts. parallel-rsync deploy files files to all hosts you listed. OPTIONS
-r --recursive recusively copy directories (OPTIONAL) -a ----archive use rsync -a (archive mode) (OPTIONAL) -z --compress use rsync compression (OPTIONAL) -h --hosts hosts file (each line "host[:port] [user]") -l --user username (OPTIONAL) -p --par max number of parallel threads (OPTIONAL) -o --outdir output directory for stdout files (OPTIONAL) -e --errdir output directory for stderr files (OPTIONAL) -t --timeout timeout (secs) (-1 = no timeout) per host (OPTIONAL) -O --options SSH options (OPTIONAL) -v --verbose turn on warning and diagnostic messages (OPTIONAL) EXAMPLE
# parallel-rsync -r -h hosts.txt -l irb2 foo /home/irb2/foo ENVIRONMENT
All four programs take similar sets of options. All of these options can be set using the following environment variables: o PSSH_HOSTS o PSSH_USER o PSSH_PAR o PSSH_OUTDIR o PSSH_VERBOSE o PSSH_OPTIONS SEE ALSO
parallel-ssh(1), parallel-scp(1), parallel-slurp(1), parallel-nuke(1), ssh(1), rsync(1) AUTHOR
Brent N. Chun <bnc@theether.org> COPYING
Copyright: 2003, 2004, 2005, 2006, 2007 Brent N. Chun NOTES
1. bnc@theether.org mailto:bnc@theether.org 03/30/2009 PARALLEL-RSYNC(1)