UNIX for Dummies Questions & Answers

Hello....

I have two servers, one has an empty / and the other has a subdirectory with a large number (4 gig) with many, many files. I need a way to transfer the files en masse from the server with the large number of files to the one that is essentially blank.

I don't have space on the used host to simply gzip all the files. I've googled this and see that there may be some combination of tar and/or gzip that will let me do this with some sort of redirection.

I really need and example line of how this can be accomplished. If my explanation seems rather sparse, I can supply more details.

Thanks!

Blaine

You can only run so many streams over the net until you saturate it. Sending compressed streams is good, and long ones. Writing any extra files is evil wasted disk I/O bandwidth, never mind the space.

I would use my xdemux tool with find to divide the file names found into N parallel streams, where n is 2-4 times the cpu core/thread count of the sending system. Then use rsh to leap the gap and cpio to pack and unpack. You can use ssh/ssh2, eating more cpu to crypt/decrypt, plus optionally get the compression in the same process, which might be slower and harder to control. The gzip might work better than compress, if you have lots of sending CPU and relatively low net speed, and vice versa. There is a gap between compress (16 bit LZ)and gzip -1 in both speed and compression. The bzip2 is almost always too slow, and rzip demands seekable files. It might work best to lay down empty directories first, so cpio does not create them with the wrong permissions.

(
find ... -type d
find ... -type l
find ... -type p
find ... -type f
)|xdemux 16 'cpio -oaH crc | gzip -9 |rsh other_host "gzcat | cpio -idmH crc" '

https://www.unix.com/shell-programmin...data-file.html

Now, if rsh won't bridge the root id, you might need to add a named pipe in each stream, where root has find to N instances that write the cpio data to the named pipe, not-root gzips and rsh's and gzcat's between named pipes, and root over there invokes N instances of cpio reading from that named pipe. It's a bit manual, but fairly secure. The named pipes can be owned by not-root accessible to himself only, and root should be able to use them because root is root, but I have no system to tinker with and check that. I guess root can group permit root in on a root-exclusive group.

My named pipes are a bit tricky: you want to be spawning N read openers first else all the writers data goes to the first reader! Not my usual medium! Funny command, too, infix pipe path: mknod pipe_path p

Code:

tar -cBpf - /dir | rsh host -l user "tar -xBpf - "

DGPickett,

Thanks for your help... I was looking for a simple, single command line script I could pipe or something. Bandwidth is not a problem. Nor is CPU, etc. I have complete control of both hosts. Also, the multiple files are in multiple subdirectories as well.
Thanks for your help! Blaine

---------- Post updated at 02:40 PM ---------- Previous update was at 02:37 PM ----------

citaylor, Thanks, this is better... will this line pick up all the files in all the subdirectories?

Also, can I use ssh instead of rsh or is rsh faster? I don't need the security that ssh gives...

Thanks again for all your assistance and responses... Blaine

If you have enough CPU horsepower(on both ends!) ssh might not be that bad.

Corona688,

Thanks! I'll keep that in mind! I'm still trying to figure out a single line of script to accomplish this task.

Thanks again! Blaine

---------- Post updated at 03:51 PM ---------- Previous update was at 02:48 PM ----------

Did I mention I'm trying to do this by close of business this Friday?

Thanks!

Blaine

Yes this will do a recursive copy from "dir". Using ssh it would be:

Code:

tar -cBpf - /dir | ssh user@host "tar -xBpf - "

If you need to relocate the dir then cd into the parent dir and type:

Code:

tar -cBpf - dir | ssh user@host "cd /anotherdir ; tar -xBpf - "

(If it is an HP-UX host you have to miss out the "B" option.)

cheers