AIX

Looking at the performance hit on my server, does it matter wich command I run?

Code:

client # rsh server tar –cf - . | tar –cv –f –

or

Code:

server # tar –cf – . | rsh client ‘cd target && tar –xv -f –‘

I think it doesn't really matter because both command strings involve a tar being run on the server and client. Both commandstrings do a read on the server and a write on the client.

Are there compelling reasons one would choose for option 1 or option 2?

--Peter

They are identical and flawed. I like '*' or '.??* *' over '.'; all file names 2 bytes shorter!

Option 3 is to put in compression, and, although newer tar's have compression options, for speed and versatility outside tar so you can use more CPUs and try different compressors: compress (faster with most output), gzip, bzip2 (slowest with least output) with various options. Usually, bzip2 is too slow to be useful, but compress, gzip -1, gzip -7 or gzip -9 wins.

Here is with compress/zcat and tar:

Code:

rsh -n server "cd /xxx ; tar cf - * | compress" | zcat | tar xf -

Networks are rarely faster than CPUs, and the gap is widening.

I also like cpio over tar, but some tar's with some options have stopped padding every file to record length with nulls! Also, you can run parallel streams to get a little more out of the network by dividing the file name stream into cpio using xdemux.c: https://www.unix.com/shell-programming-scripting/146458-sort-big-data-file.html After all, what are 32 CPU systems for? Parallel streams means you can use slower, better compression and still max out the network, getting a higher total bandwidth. Of course, if one file is 90% of the bulk, not gonna help.

Code:

find * -type f | xdemux 4 'cpio -oaH crc | gzip -9 | rsh client "cd target ; gzcat | cpio -idmH crc" '

There is the more secure ssh, if security ever becomes a concern, but the encryption costs some speed. It has optional gzip-like -C compression option, if you want to save a few keystrokes and lose some speed.

Also, if there are already old files there, the -u can save you some writing of identical files. That might not be your bottleneck, and someone might play with the timestamps on files, or have updates you do not want to preserve, but sometimes updated files are worth keeping (like local configuration).

Finally, -n saves a few microseconds in forking an unused process for full duplex, when rsh reads nothing from stdin.