Transfer large number of files host to host

Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Transfer large number of files host to host
# 1  
Old 10-20-2010
Transfer large number of files host to host


I have two servers, one has an empty / and the other has a subdirectory with a large number (4 gig) with many, many files. I need a way to transfer the files en masse from the server with the large number of files to the one that is essentially blank.

I don't have space on the used host to simply gzip all the files. I've googled this and see that there may be some combination of tar and/or gzip that will let me do this with some sort of redirection.

I really need and example line of how this can be accomplished. If my explanation seems rather sparse, I can supply more details.


# 2  
Old 10-20-2010
You can only run so many streams over the net until you saturate it. Sending compressed streams is good, and long ones. Writing any extra files is evil wasted disk I/O bandwidth, never mind the space.

I would use my xdemux tool with find to divide the file names found into N parallel streams, where n is 2-4 times the cpu core/thread count of the sending system. Then use rsh to leap the gap and cpio to pack and unpack. You can use ssh/ssh2, eating more cpu to crypt/decrypt, plus optionally get the compression in the same process, which might be slower and harder to control. The gzip might work better than compress, if you have lots of sending CPU and relatively low net speed, and vice versa. There is a gap between compress (16 bit LZ)and gzip -1 in both speed and compression. The bzip2 is almost always too slow, and rzip demands seekable files. It might work best to lay down empty directories first, so cpio does not create them with the wrong permissions.

find ... -type d
find ... -type l
find ... -type p
find ... -type f
)|xdemux 16 'cpio -oaH crc | gzip -9 |rsh other_host "gzcat | cpio -idmH crc" '

Now, if rsh won't bridge the root id, you might need to add a named pipe in each stream, where root has find to N instances that write the cpio data to the named pipe, not-root gzips and rsh's and gzcat's between named pipes, and root over there invokes N instances of cpio reading from that named pipe. It's a bit manual, but fairly secure. The named pipes can be owned by not-root accessible to himself only, and root should be able to use them because root is root, but I have no system to tinker with and check that. I guess root can group permit root in on a root-exclusive group.

My named pipes are a bit tricky: you want to be spawning N read openers first else all the writers data goes to the first reader! Not my usual medium! Funny command, too, infix pipe path: mknod pipe_path p

Last edited by DGPickett; 10-20-2010 at 05:53 PM..
# 3  
Old 10-20-2010
tar -cBpf - /dir | rsh host -l user "tar -xBpf - "

This User Gave Thanks to citaylor For This Post:
# 4  
Old 10-20-2010

Thanks for your help... I was looking for a simple, single command line script I could pipe or something. Bandwidth is not a problem. Nor is CPU, etc. I have complete control of both hosts. Also, the multiple files are in multiple subdirectories as well.
Thanks for your help! Blaine

---------- Post updated at 02:40 PM ---------- Previous update was at 02:37 PM ----------

citaylor, Thanks, this is better... will this line pick up all the files in all the subdirectories?

Also, can I use ssh instead of rsh or is rsh faster? I don't need the security that ssh gives...

Thanks again for all your assistance and responses... Blaine
# 5  
Old 10-20-2010
If you have enough CPU horsepower(on both ends!) ssh might not be that bad.
# 6  
Old 10-20-2010

Thanks! I'll keep that in mind! I'm still trying to figure out a single line of script to accomplish this task.

Thanks again! Blaine

---------- Post updated at 03:51 PM ---------- Previous update was at 02:48 PM ----------

Did I mention I'm trying to do this by close of business this Friday?


# 7  
Old 10-21-2010
Yes this will do a recursive copy from "dir". Using ssh it would be:
tar -cBpf - /dir | ssh user@host "tar -xBpf - "

If you need to relocate the dir then cd into the parent dir and type:
tar -cBpf - dir | ssh user@host "cd /anotherdir ; tar -xBpf - "

(If it is an HP-UX host you have to miss out the "B" option.)

This User Gave Thanks to citaylor For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Connect to target host from Source host.

Hi All, Need to connect to target host and execute a command and connect back to source host to continue with next set of command execution. Is there a utility/command using which we can connect to target host ? Please suggest. Note: Netezza database is instaled on Linux server. ... (4 Replies)
Discussion started by: Nagaraja Akkiva
4 Replies

2. Solaris

Need to recover/move diskgroup from failed host to another host

Hi All I am having VxVm on two Solaris hosts. host1 is using disk group dgHR. right now this server went down due to hardware fault. Not I need to import this dgHR into host2 server. Please let me know the procedure for the same. (1 Reply)
Discussion started by: amity
1 Replies

3. Shell Programming and Scripting

How to pass a variable from one host to another host

Hi , I have to pass a string through a variable from one host to another. Is that any any way to do it . Please help me (4 Replies)
Discussion started by: arukuku
4 Replies

4. IP Networking

ping can not recognize host but host command can

Hi, I have a weird problem. when ever I do ping command like for example ping unix.comI get the following message: # ping ping: unknown host but when I use host the computer is able to know the host. # host has address mail is... (2 Replies)
Discussion started by: programAngel
2 Replies

5. UNIX for Advanced & Expert Users

Help! How to find the local host after few ssh hops to remote host???

I do a ssh to remote host(A1) from local host(L1). I then ssh to another remote(A2) from A1. When I do a who -m from A2, I see the "connected from" as "A1". => who -m userid pts/2 2010-03-27 08:47 (A1) I want to identify who is the local host who initiated the connection to... (3 Replies)
Discussion started by: gomes1333
3 Replies

6. Shell Programming and Scripting

running commands to remote host from centralized host

Gurus/Experts We have a centralized UNIX/Solaris server from where we can actually ssh to all other UNIX/Solaris servers...I need to write a script that reside on this centerlized server and do FileSystem monitoring (basically run df -h or -k) of other remote servers and then send an email to me... (6 Replies)
Discussion started by: anjum.suri
6 Replies

7. Solaris

How to delete the files from local host to remote host

Hi all, i am copying .gz files from production server to development server using "scp" requirement is after copying .gz files i want to delete old .gz files(two days back) in development server from production server. like this way i need to delelte .log ,.z and .dmp files... (3 Replies)
Discussion started by: krishna176
3 Replies

8. Solaris

Tar files, transfer to remote host and delelte source

Hi, Im having a problem here. I have a directory that holds many subdirs and files (by many I mean MANY thousands). What I want to do is make blocks of these files and transfer them block by block to a remote host; but once a given file is already placed in the remote host, the script must... (0 Replies)
Discussion started by: Dago
0 Replies

9. UNIX for Advanced & Expert Users

host alias not working: host not found

Hello, I am working on HP-UX , and in the /etc/hosts file we have setup an alias: devmach2 devma2v The alias devma2v does not work. Error when pinging devma2v ping: unknown host devma2v For devmach2 the ping works fine , returning the correct IP... (4 Replies)
Discussion started by: FunnyCats
4 Replies

10. IP Networking

QNX host cannot ping SCO host, vice versa

The problem I am facing now is that the QNX host could not ping the SCO host and vice versa. They are in the same domain, ie, 172.20.3.xx. As I am very new to Unix, I guess I must have missed out some important steps. Pls help... Thanx alot (2 Replies)
Discussion started by: gavon
2 Replies
Login or Register to Ask a Question

Featured Tech Videos