Sponsored Content
Top Forums Shell Programming and Scripting How to copy very large directory trees Post 302730001 by Chubler_XL on Monday 12th of November 2012 04:56:14 AM
Old 11-12-2012
Rsync will skip any files that are already in the destination by default it uses file size+date+time or with the -c option computes a checksum.

--partial saves a partially transfered file and will resume from where it was, without this, a file that is in being transfered when a server is rebooted or goes off the network will be deleted and the file re-transfered. This can be important if large files are being transfered and starting again will waste a lot of bandwidth.

--del can be important to remove files on the destination side that don't exist on the source side.

Try is something like this to copy a directory path keeping owner/permissions and removing and file in dest not in source

Code:
rsync -avc --del --partial /path/to/source/files/ dest:/path/to/dest/files

also consider something like --bwlimit=4000 to avoid swamping your network by limiting the transfer to 4Mbps.
 

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Copy large dump file

Hi Experts.. Could anyone please let me know the easier way to copy large dump of files from one server to another. I am trying to copy a set of dump files on two different servers placed in different geographic locations.. Though there are other factors such as latency, etc., slowing up the... (4 Replies)
Discussion started by: ganga.dharan
4 Replies

2. Shell Programming and Scripting

Best way to diff two huge directory trees

Hi I have a job that will be running nightly incremental backsup of a large directory tree. I did the initial backup, now I want to write a script to verify that all the files were transferred correctly. I did something like this which works in principle on small trees: diff -r -q... (6 Replies)
Discussion started by: same1290
6 Replies

3. UNIX for Advanced & Expert Users

How to rsync or tar directory trees, with hidden directory, but without files?

I want to backup all the directory tress, including hidden directories, without copying any files. find . -type d gives the perfect list. When I tried tar, it won't work for me because it tars all the files. find . -type d | xargs tar -cvf a.tar So i tried rsync. On my own test box, the... (4 Replies)
Discussion started by: fld2007
4 Replies

4. Shell Programming and Scripting

how to copy the directory but not copy certain file

Hi experts cp bin root src /mnt but not copy bin/bigfile any help? ( I post this thread in the "redhat" forum wrongly, I don't know how to withdraw that question in that wrong forum) Thanks (6 Replies)
Discussion started by: yanglei_fage
6 Replies

5. UNIX for Dummies Questions & Answers

Is it better/possible to pause the rsyncing of a very large directory?

Possibly a dumb question, but I'm deciding how I'm going to do this. I'm currently rsyncing a 25TB directory (with several layers of sub directories most of which have video files ranging from 500 megs to 4-5 gigs), from one NAS to another using rsync -av. By the time I need to act ~15TB should... (3 Replies)
Discussion started by: DeCoTwc
3 Replies

6. Slackware

What is the medium usually used to backup large trees?

Hi: What's asked. (2 Replies)
Discussion started by: stf92
2 Replies

7. Shell Programming and Scripting

Copy the files in directory and sub folders as it is to another directory.

How to copy files from one directory to another directory with the subfolders copied. If i have folder1/sub1/sub2/* it needs to copy files to folder2/sub1/sub2/*. I do not want to create sub folders in folder2. Can copy command create them automatically? I tried cp -a and cp -R but did... (4 Replies)
Discussion started by: santosh2626
4 Replies

8. Shell Programming and Scripting

ksh - Checking directory trees containing wild cards

Hi Can somebody please show me how to check from within a KSH script if a directory exists on that same host when parts of the directory tree are unknown? If these wildcard dirs were the only dirs at that level then ... RETCODE=$(ls -l /u01/app/oracle/local/*/* | grep target_dir) ... will... (4 Replies)
Discussion started by: user052009
4 Replies
icmstart.rc(7)						      icmstart resource file						    icmstart.rc(7)

NAME
icmstart.rc - The icmstart(1) resource file DESCRIPTION
The icmstart.rc file defines the initial files of a program that are installed by icmstart(1). By default icmstart.rc is found in /etc/icmake/icmstart.rc, but if a file $HOME/.icmake/icmstart.rc is found then the latter file is used by icmstart(1). $HOME/.icmake/icmstart.rc in turn is overruled by specifying a -c option when invoking icmstart(1). The default resource file contains the following specifications, preparing for the construction of a C++ program using icmbuild(1): CLASSES P main.cc P main.ih P usage.cc P version.cc P ? scanner P ? parser The icmstart.rc file may contain: o empty lines, which are ignored; o lines beginning with a hash-character (#), also ignored; o Optionally: a P (don't install with icmstart xxx library) or an L (don't install with icmstart xxx program), followed by a blank; see also icmstart(1). o Optionall: a ? (followed by a blank) following the optional P or L in which case the installation of the element must be confirmed by the user. o Source path names. Relative path names are interpreted relative to the current working directory. Environment variables are not expanded, the initial tilde character is interpreset as the user's home directory. o Source path names followed by a destination (relative to the directory initialized by icmstart(1)). A source path may be either a file or a directory. If a directory is specified then all files in and below that directory will be installed. When specifying a destination a path may be specified as well. However, the destination path is always interpreted relative to the instal- lation directory specified when calling icmstart(1). If a destination is omitted, the source is installed as specified below icmstart(1)'s destination directory. EXAMPLES
The following entries can be used in an icmstart.rc file: o source `Source' must be in the skeleton directory visited by icmstart(1) and is installed as source in the destination directory specified when calling icmstart(1). o source dest `Source' must be in the skeleton directory visited by icmstart(1) and is installed as dest in the destination directory specified when calling icmstart(1). o ~/path `~/path' is installed as ~/path in the destination directory specified when calling icmstart(1). The ~-character is expanded to the user's home directory. o ~/path dest `~/path' is installed as dest in the destination directory specified when calling icmstart(1). The ~-characteris expanded to the user's home directory. o /path `/path' is installed as ./path in the destination directory specified when calling icmstart(1). o /path dest `/path' is installed as dest in the destination directory o ? source The user must confirm the installation of source. o ? source dest The user must confirm the installation of source as dest in the destination directory specified when calling icmstart(1). o # text Interpreted as comment and ignored. The #-character must be the first character on the line. o an empty line Empty lines are skipped. FILES
The mentioned paths are sugestive only and may be installation dependent: o /usr/share/icmake/icmconf Example of an icmbuild configuration file; o /usr/share/icmake/CLASSES Example of an icmbuild CLASSES file. o /usr/share/icmake/icmstart.rc Default skeleton resource file. SEE ALSO
icmake(1), icmbuild(1), icmconf(7), icmstart(1) BUGS
None reported COPYRIGHT
This is free software, distributed under the terms of the GNU General Public License (GPL). AUTHOR
Frank B. Brokken (f.b.brokken@rug.nl). icmake.7.18.00.tar.gz 1992-2012 icmstart.rc(7)
All times are GMT -4. The time now is 07:31 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy