Sponsored Content
Top Forums Shell Programming and Scripting scp or rsync multiple files in parallel from a remote host Post 302682621 by jim mcnamara on Monday 6th of August 2012 01:50:50 PM
Old 08-06-2012
This will try to send all of the files:
Code:
RHOST=srv245005
CCMD="ls /db/remote_backup_area"
 
for BackupFile in 'ssh -q oracle@${RHOST} ${CCMD}'; do
        echo "attempting to copy file: " $BackupFile
 
        scp -pq $hostname:/db/remote_backup_area/$file /db/local_dump_area &
 
        # or use rsync

done
wait
exit

which can swamp the connection. This sends 10 files at a go

Code:

cnt=1
RHOST=srv245005
CCMD="ls /db/remote_backup_area"
 
for BackupFile in 'ssh -q oracle@${RHOST} ${CCMD}'; do
        echo "attempting to copy file: " $BackupFile
         
        scp -pq $hostname:/db/remote_backup_area/$file /db/local_dump_area &
        cnt=$(( $cnt + 1 ))
        if [ cnt -eq 10 ] then
           wait
           cnt=1
        fi

done
wait
exit

Change the "10" to something that is reasonable

If you want some kind of TTY ouput you cannot do the above, you have to write to a file on the scp command line.
Code:
( 
  echo "starting $file `date`" >/tmp/${file}.log
  scp -pq $hostname:/db/remote_backup_area/$file /db/local_dump_area 2>>/tmp/${file}.log
  echo "done $file `date`" >>/tmp/${file}.log
) &

This mess replaces the one line scp command in the earlier examples. And creates lots of logfiles.

Last edited by jim mcnamara; 08-06-2012 at 02:57 PM..
This User Gave Thanks to jim mcnamara For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Backing up files to a remote host question.

I need to know how to successfully back up(and compress) files from a local machine to a remote host. Will this work? tar -cvf backup.tar -C /user/somedir | gzip backup.tar | rsh some.domain.com/user/somedir thanks in advance! (3 Replies)
Discussion started by: WeNdeL
3 Replies

2. Solaris

How to delete the files from local host to remote host

Hi all, i am copying .gz files from production server to development server using "scp" command.my requirement is after copying .gz files i want to delete old .gz files(two days back) in development server from production server. like this way i need to delelte .log ,.z and .dmp files... (3 Replies)
Discussion started by: krishna176
3 Replies

3. Shell Programming and Scripting

rsync multiple remote directories one connection

Hi, I want to specify multiple remote directories but want to transfer them in a single command with one connection with remote server. e.g. rsync -vrt --size-only --delete user@host:/home/user1/dir1 user@host:/home/user1/dir2 user@host:/home/user1/dir3 .... local_dir/ I want to... (0 Replies)
Discussion started by: sardare
0 Replies

4. Shell Programming and Scripting

rsync multiple directories into single directory on remote

I would like to find and backup all *.mp4 files from /Pictures and its sub-directories and move them to a single directory on a remote. I can find and move the files but I don't want the directory structure...just the files to be placed in a single remote directory. To find my files I use ... (1 Reply)
Discussion started by: wreckedred
1 Replies

5. Shell Programming and Scripting

scp files that are 3 days older from remote server-

hello, i am trying to get a list of files to be scped from the remote server by running the below in my local unix server ( note - there is a passwordless connectivity setup made between the local and remote server) and, we use KSH. --- ssh $scp_host "find /a/b/c/*/ -iname "$remote_file"" >... (4 Replies)
Discussion started by: billpeter3010
4 Replies

6. Shell Programming and Scripting

Pause processes in remote host and resume execution in another remote host

Hi, Given addresses of 2 remote machines, using a shell script is it possible to get the state of running processes in "src" stop all the processes in "src" exit out of "src" ssh into "dest" resume the state of executing processes captured in step 1 in "dest" Assumption: "src" is... (3 Replies)
Discussion started by: Saeya Darsan
3 Replies

7. Shell Programming and Scripting

How can we overcome Broken pipe error during scp,SFTP,Rsync while transferring big files.?

Hello All, Hope all are doing well. We use scp (some times sftp and rsync also) for transferring big files (around 2GB each ) from 1 Network to another Network. The Issues which we face :- During transfer some times( Once in 1 week (or twice)) , the speed of transfer gets down to 30 kb/s,... (2 Replies)
Discussion started by: Upendra Bhushan
2 Replies

8. UNIX for Beginners Questions & Answers

scp of multiple files to remote server

Hello, I would like to scp multiple files to a remote server that requires a password for the process to be completed. I have 30 folders (x_1, x_2 ... x_30), each containing 25 files. What I want to do is scp 1 out of the 25 files (file called bvals) for all my folders to a remote server and... (3 Replies)
Discussion started by: nasia.m
3 Replies

9. UNIX for Beginners Questions & Answers

How to paste multiple files in parallel?

Hi all, I am trying to paste thousands of files together into a matrix. Each file has only 1 column and all the files have the same number of rows (~27k rows). I tried paste * > output as well as some other for loops but the output only contains the columns from the 1st and last files. The... (8 Replies)
Discussion started by: notimenocall
8 Replies

10. Shell Programming and Scripting

Copy local files to single remote host but multiple folders using rsync

I'm trying to copy a file myfile.scr from my local Linux server to multiple folders on remote AiX server using single rsync command. Below command helps me copy the file "myfile.scr" from my localhost to a remote host folder "/app/deployment/tmpfiles" rsync --delay-updates -F --compress... (1 Reply)
Discussion started by: mohtashims
1 Replies
SCP(1)							    BSD General Commands Manual 						    SCP(1)

NAME
scp -- secure copy (remote file copy program) SYNOPSIS
scp [-12346BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file] [-l limit] [-o ssh_option] [-P port] [-S program] [[user@]host1:]file1 ... [[user@]host2:]file2 DESCRIPTION
scp copies files between hosts on a network. It uses ssh(1) for data transfer, and uses the same authentication and provides the same secu- rity as ssh(1). Unlike rcp(1), scp will ask for passwords or passphrases if they are needed for authentication. File names may contain a user and host specification to indicate that the file is to be copied to/from that host. Local file names can be made explicit using absolute or relative pathnames to avoid scp treating file names containing ':' as host specifiers. Copies between two remote hosts are also permitted. The options are as follows: -1 Forces scp to use protocol 1. -2 Forces scp to use protocol 2. -3 Copies between two remote hosts are transferred through the local host. Without this option the data is copied directly between the two remote hosts. Note that this option disables the progress meter. -4 Forces scp to use IPv4 addresses only. -6 Forces scp to use IPv6 addresses only. -B Selects batch mode (prevents asking for passwords or passphrases). -C Compression enable. Passes the -C flag to ssh(1) to enable compression. -c cipher Selects the cipher to use for encrypting the data transfer. This option is directly passed to ssh(1). -F ssh_config Specifies an alternative per-user configuration file for ssh. This option is directly passed to ssh(1). -i identity_file Selects the file from which the identity (private key) for public key authentication is read. This option is directly passed to ssh(1). -l limit Limits the used bandwidth, specified in Kbit/s. -o ssh_option Can be used to pass options to ssh in the format used in ssh_config(5). This is useful for specifying options for which there is no separate scp command-line flag. For full details of the options listed below, and their possible values, see ssh_config(5). AddressFamily BatchMode BindAddress ChallengeResponseAuthentication CheckHostIP Cipher Ciphers Compression CompressionLevel ConnectionAttempts ConnectTimeout ControlMaster ControlPath GlobalKnownHostsFile GSSAPIAuthentication GSSAPIDelegateCredentials HashKnownHosts Host HostbasedAuthentication HostKeyAlgorithms HostKeyAlias HostName IdentityFile IdentitiesOnly IPQoS KbdInteractiveDevices KexAlgorithms LogLevel MACs NoHostAuthenticationForLocalhost NumberOfPasswordPrompts PasswordAuthentication PKCS11Provider Port PreferredAuthentications Protocol ProxyCommand PubkeyAuthentication RekeyLimit RhostsRSAAuthentication RSAAuthentication SendEnv ServerAliveInterval ServerAliveCountMax StrictHostKeyChecking TCPKeepAlive UsePrivilegedPort User UserKnownHostsFile VerifyHostKeyDNS -P port Specifies the port to connect to on the remote host. Note that this option is written with a capital 'P', because -p is already reserved for preserving the times and modes of the file in rcp(1). -p Preserves modification times, access times, and modes from the original file. -q Quiet mode: disables the progress meter as well as warning and diagnostic messages from ssh(1). -r Recursively copy entire directories. Note that scp follows symbolic links encountered in the tree traversal. -S program Name of program to use for the encrypted connection. The program must understand ssh(1) options. -v Verbose mode. Causes scp and ssh(1) to print debugging messages about their progress. This is helpful in debugging connection, authentication, and configuration problems. EXIT STATUS
The scp utility exits 0 on success, and >0 if an error occurs. SEE ALSO
rcp(1), sftp(1), ssh(1), ssh-add(1), ssh-agent(1), ssh-keygen(1), ssh_config(5), sshd(8) HISTORY
scp is based on the rcp(1) program in BSD source code from the Regents of the University of California. AUTHORS
Timo Rinne <tri@iki.fi> Tatu Ylonen <ylo@cs.hut.fi> BSD
December 9, 2010 BSD
All times are GMT -4. The time now is 07:23 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy