UNIX for Advanced & Expert Users

TAR AND CPIO not quick enough - OS LEVEL 5.2

We need to copy millions of files around 42gb. Ls command does not come back complaining of memory.
A basic copy comes back with i cant read the directory. TAR too slow, left running for 48 hours but killed as users were accessing the system.
Does anyone know if there is a command that can copy this data quickly or is there a script that could be written to assist.

Any help appreciated

I doubt there is a faster tool. If this is a repeated task, maybe think about using the snapshot command to split off a snapshot and get a backup of those files so your production can run undisturbed meanwhile.

We assume you are copying disc-to-disc within the same computer.
Disc mirroring or snapshot solutions are going to be fastest - depends on what type of filesystem this is and whether you have spare discs.


First find out if we can process this filesystem at all and get some stats:
# How big and how many inodes?
df -k /filesystem
# Can we access every inode? Is the number of files similar to "df" ?
find /filesystem -xdev -depth -print | wc -l



Of the unix commands the best combination is "find" piped to "cpio -p". See the "man" pages for both commands. Some versions of "cpio" will not work for large files (check this first on your system).

The copy should not be to a directory under the master. For speed this works best if the copy is on a different filesystem.

Code:

# Example - adjust for your situation and your versions of "find" and "cpio"
master_dir="/master_dir_name"
copy_dir="/copy_dir_name"
cd "${master_dir}"
# Omit -v switch after testing because filename display slows the process.
find . -xdev -depth -print | cpio -pdumv "${copy_dir}"