Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

scp or rsync multiple files in parallel from a remote host

Shell Programming and Scripting


Tags
korn shell scripting, rsync, scp

Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 08-06-2012   -   Original Discussion by Big_Jeffrey
Big_Jeffrey's Unix or Linux Image
Big_Jeffrey Big_Jeffrey is offline
Registered User
 
Join Date: Aug 2012
Last Activity: 30 October 2014, 10:12 AM EDT
Posts: 4
Thanks: 3
Thanked 0 Times in 0 Posts
scp or rsync multiple files in parallel from a remote host

Hi.

I'm trying to speed up an rsync command by running it in parallel. There's no real option for this other than if the files are in multiple directories (which they're not). And even then there's no way of knowing if rsync has succeeded as the process is running in the background .. and there's no return code from the process id. Any ideas?

I thought about just using scp but scripting it is proving trickier than I thought and I'm getting errors on the for loop when I try to introduce variables into the ssh part:

E.g.


Code:
RHOST=srv245005
CCMD="ls /db/remote_backup_area"
 
for BackupFile in 'ssh -q oracle@${RHOST} ${CCMD}'; do
        echo "attempting to copy file: " $BackupFile
 
        scp -p $hostname:/db/remote_backup_area/$file /db/local_dump_area
 
        # or use rsync
        #rsync -av -e ssh oracle@${RHOST}:/db/remote_backup_area/${BackupFile} /db/local_dump_area"

done
exit

Any help much appreciated.
Moderator's Comments:
scp or rsync multiple files in parallel from a remote host code tags please

Last edited by jim mcnamara; 08-06-2012 at 02:47 PM..
Sponsored Links
    #2  
Old Unix and Linux 08-06-2012   -   Original Discussion by Big_Jeffrey
jim mcnamara's Unix or Linux Image
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 23 February 2018, 11:46 AM EST
Location: NM
Posts: 11,306
Thanks: 587
Thanked 1,136 Times in 1,046 Posts
This will try to send all of the files:


Code:
RHOST=srv245005
CCMD="ls /db/remote_backup_area"
 
for BackupFile in 'ssh -q oracle@${RHOST} ${CCMD}'; do
        echo "attempting to copy file: " $BackupFile
 
        scp -pq $hostname:/db/remote_backup_area/$file /db/local_dump_area &
 
        # or use rsync

done
wait
exit

which can swamp the connection. This sends 10 files at a go



Code:

cnt=1
RHOST=srv245005
CCMD="ls /db/remote_backup_area"
 
for BackupFile in 'ssh -q oracle@${RHOST} ${CCMD}'; do
        echo "attempting to copy file: " $BackupFile
         
        scp -pq $hostname:/db/remote_backup_area/$file /db/local_dump_area &
        cnt=$(( $cnt + 1 ))
        if [ cnt -eq 10 ] then
           wait
           cnt=1
        fi

done
wait
exit

Change the "10" to something that is reasonable

If you want some kind of TTY ouput you cannot do the above, you have to write to a file on the scp command line.


Code:
( 
  echo "starting $file `date`" >/tmp/${file}.log
  scp -pq $hostname:/db/remote_backup_area/$file /db/local_dump_area 2>>/tmp/${file}.log
  echo "done $file `date`" >>/tmp/${file}.log
) &

This mess replaces the one line scp command in the earlier examples. And creates lots of logfiles.

Last edited by jim mcnamara; 08-06-2012 at 02:57 PM..
The Following User Says Thank You to jim mcnamara For This Useful Post:
Big_Jeffrey (08-07-2012)
Sponsored Links
    #3  
Old Unix and Linux 08-06-2012   -   Original Discussion by Big_Jeffrey
Big_Jeffrey's Unix or Linux Image
Big_Jeffrey Big_Jeffrey is offline
Registered User
 
Join Date: Aug 2012
Last Activity: 30 October 2014, 10:12 AM EDT
Posts: 4
Thanks: 3
Thanked 0 Times in 0 Posts
Hi Jim. Thanks for the reply. Did the above code work for you? I get an 'issue' in the first line of the for loop. It wont read the value of the $RHOST and $CCMD variables that are part of the loop condition .. so it does nothing unfortunately. If I hard code them in then it works .. I'm missing something from my code.
    #4  
Old Unix and Linux 08-06-2012   -   Original Discussion by Big_Jeffrey
244an's Unix or Linux Image
244an 244an is offline
Registered User
 
Join Date: Jul 2012
Last Activity: 7 October 2014, 7:14 PM EDT
Posts: 71
Thanks: 9
Thanked 13 Times in 13 Posts
Shouldn't you have backticks in the for-line?


Code:
 for BackupFile in `ssh -q oracle@${RHOST} ${CCMD}`; do

The Following User Says Thank You to 244an For This Useful Post:
Big_Jeffrey (08-07-2012)
Sponsored Links
    #5  
Old Unix and Linux 08-07-2012   -   Original Discussion by Big_Jeffrey
Chubler_XL's Unix or Linux Image
Chubler_XL Chubler_XL is offline Forum Staff  
Moderator
 
Join Date: Oct 2010
Last Activity: 22 February 2018, 10:31 PM EST
Posts: 3,466
Thanks: 151
Thanked 1,226 Times in 1,125 Posts
I find it hard to believe that even 1 rsync will not swamp your bandwidth as disk I/O should be WAY faster than the network.

If you want more speed try the -z (or --compress) rsync option and compress the data transfered.

Funny thing is I usually find that rsync swamps my network and even web service requests from/to the boxes concerned tend to timeout a lot. More often than not, I find myself using --bwlimit to slow things down!
The Following User Says Thank You to Chubler_XL For This Useful Post:
Big_Jeffrey (08-07-2012)
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
rsync multiple directories into single directory on remote wreckedred Shell Programming and Scripting 1 01-31-2011 05:58 PM
rsync multiple remote directories one connection sardare Shell Programming and Scripting 0 03-11-2010 01:44 AM
How to delete the files from local host to remote host krishna176 Solaris 3 03-24-2007 05:48 PM
Tar files, transfer to remote host and delelte source Dago Solaris 0 02-28-2006 05:35 PM
Backing up files to a remote host question. WeNdeL UNIX for Dummies Questions & Answers 3 10-02-2002 12:14 PM



All times are GMT -4. The time now is 04:22 PM.