Solaris

User needs to copy a number of directories. Currently, they are using cp -pr which is taking way too long.

Wonder if its worth tarring and compressing first?

Any recommendations for what would be quickest way?

Did you search the forums before posting?

For example:

The Fastest for copy huge data

The answer will always be 'It depends.'

What can you tell us about the server(s) and data? It will probably be trial and error.

If this is a regular process, consider rsync which will only copy differences. It might still be slower if you have milliions over small files though. If there is lots of data change, perhaps you should consider snapshots, but then you haven't told us what OS you have, so i don't know if that's available to you in any useful way.





Robin

First, a sanity check: 4 terabytes, divided by a good spinning-disk transfer rate of 100 megabytes per second, is ten hours at minimum. Easily 20 for more average disks. You'd need a fancy striped RAID array, or an implausibly large SSD, to beat that.

A great deal depends on your data of course, but knowing why trillions of teeny files are slower than fewer, larger files is not helpful in making the system access trillions of teeny files faster.

Generally speaking? cp does not have a "go faster" button, or else we'd be pushing it all the time anyway. cp is not slow, naive, or wasteful.

Do it on a block level if you can.

Use zfs send / receive using netcat (the data stream is not encrypted over network).
It really versatile, poor man enterprise level file system replication, features matching expensive storage arrays.

For local stuff, mirror the pool and use zpool split.
If you are using SVM, mirror, remove from mirror, use the copy.

Using cp/rsync on large number of files is bound to take longer on same hardware.

Alternatively, you can use dd as well if you want entire disks, but it's a one time crude operation and long requires downtime on source.

Personally I prefer rsync to cp. But for 4TB, it probably doesn't make enough of a difference. Are both volumes on the same server? Are you using a SAN? You might find that SAN snapshot and replication is the best way to move the data. But that depends on your storage system. You might want to provide more information on the type of disk storage that you use and check with your SAN vendor, if you use a SAN, to see what they think.

This sounds like a good idea, worth trying. How about this to "blockify" the I/O:

Code:

cd /source ; tar cf - * | (cd /target ; tar xf - )

I hope this helps.

bakunin