scp or rsync multiple files in parallel from a remote host
Hi.
I'm trying to speed up an rsync command by running it in parallel. There's no real option for this other than if the files are in multiple directories (which they're not). And even then there's no way of knowing if rsync has succeeded as the process is running in the background .. and there's no return code from the process id. Any ideas?
I thought about just using scp but scripting it is proving trickier than I thought and I'm getting errors on the for loop when I try to introduce variables into the ssh part:
E.g.
Code:
RHOST=srv245005
CCMD="ls /db/remote_backup_area"
for BackupFile in 'ssh -q oracle@${RHOST} ${CCMD}'; do
echo "attempting to copy file: " $BackupFile
scp -p $hostname:/db/remote_backup_area/$file /db/local_dump_area
# or use rsync
#rsync -av -e ssh oracle@${RHOST}:/db/remote_backup_area/${BackupFile} /db/local_dump_area"
done
exit
Any help much appreciated.
Moderator's Comments:
code tags please
Last edited by jim mcnamara; 08-06-2012 at 02:47 PM..
I need to know how to successfully back up(and compress) files from a local machine to a remote host.
Will this work?
tar -cvf backup.tar -C /user/somedir | gzip backup.tar | rsh some.domain.com/user/somedir
thanks in advance! (3 Replies)
Hi all,
i am copying .gz files from production server to development server using
"scp" command.my requirement is after copying .gz files i want to delete old
.gz files(two days back) in development server from production server.
like this way i need to delelte .log ,.z and .dmp files... (3 Replies)
Hi,
I want to specify multiple remote directories but want to transfer them in a single command with one connection with remote server.
e.g.
rsync -vrt --size-only --delete user@host:/home/user1/dir1 user@host:/home/user1/dir2 user@host:/home/user1/dir3 .... local_dir/
I want to... (0 Replies)
I would like to find and backup all *.mp4 files from /Pictures and its sub-directories and move them to a single directory on a remote. I can find and move the files but I don't want the directory structure...just the files to be placed in a single remote directory.
To find my files I use
... (1 Reply)
hello,
i am trying to get a list of files to be scped from the remote server
by running the below in my local unix server ( note - there is a passwordless connectivity setup made between the local and remote server) and, we use KSH.
---
ssh $scp_host "find /a/b/c/*/ -iname "$remote_file"" >... (4 Replies)
Hi,
Given addresses of 2 remote machines, using a shell script is it possible to
get the state of running processes in "src"
stop all the processes in "src"
exit out of "src"
ssh into "dest"
resume the state of executing processes captured in step 1 in "dest"
Assumption:
"src" is... (3 Replies)
Hello All,
Hope all are doing well.
We use scp (some times sftp and rsync also) for transferring big files (around 2GB each ) from 1 Network to another Network.
The Issues which we face :-
During transfer some times( Once in 1 week (or twice)) , the speed of transfer gets down to 30 kb/s,... (2 Replies)
Hello,
I would like to scp multiple files to a remote server that requires a password for the process to be completed.
I have 30 folders (x_1, x_2 ... x_30), each containing 25 files. What I want to do is scp 1 out of the 25 files (file called bvals) for all my folders to a remote server and... (3 Replies)
Hi all,
I am trying to paste thousands of files together into a matrix. Each file has only 1 column and all the files have the same number of rows (~27k rows). I tried
paste * > output as well as some other for loops
but the output only contains the columns from the 1st and last files. The... (8 Replies)
I'm trying to copy a file myfile.scr from my local Linux server to multiple folders on remote AiX server using single rsync command.
Below command helps me copy the file "myfile.scr" from my localhost to a remote host folder "/app/deployment/tmpfiles"
rsync --delay-updates -F --compress... (1 Reply)
Discussion started by: mohtashims
1 Replies
LEARN ABOUT DEBIAN
vzmigrate
vzmigrate(8) Containers vzmigrate(8)NAME
vzmigrate - migrate a container between two OpenVZ servers
SYNOPSIS
vzmigrate [-r|--remove-area yes|no] [--ssh=ssh_options] [--rsync=rsync_options] [--keep-dst] [--online] [-v] destination_address CTID
DESCRIPTION
This utility is used to migrate a container from one (source) Hardware Node (HN) to another (destination) HN. The utility can migrate
either stopped or running container. For a stopped container, simple CT private area transfer is performed (rsync(1) is used for file
transfer). For running containers, migration may be offline (default) or online.
This program uses ssh as a transport layer. You will need to put ssh public key to destination node and be able to connect to node without
entering password.
OPTIONS -r, --remove-area yes | no
Whether to remove a container area on source HN for the successfully migrated container. Default is yes.
--ssh=options
Additional options that will be passed to ssh while establishing connection to destination HN.
--rsync=options
Additional options that will be passed to rsync(8). You may add options like -z to enable data compression if you are migrating
over a slow link.
--keep-dst
Do not clean synced destination container private area in case of some error. It makes sense to use this option on big container
migration to avoid syncing container private area again in case some error (on container stop for example) occurs during first
migration attempt.
--online
Perform online (zero-downtime) migration: during the migration the container hangs for a while and after the migration it continues
working as though nothing has happened.
-v Verbose mode. Causes vzmigrate to print debugging messages about its progress. Multiple -v options increase the verbosity. The
maximum is 3.
EXAMPLES
Migration of CT 101 to 192.168.1.130 with downtime:
vzmigrate 192.168.1.130 101
Online migration of CT 102 to 192.168.1.130:
vzmigrate --online 192.168.1.130 102
EXIT STATUS
0 EXIT_OK
Command completed successfully.
1 EXIT_USAGE
Bad command line options.
2 EXIT_VE_STOPPED
Container is stopped.
4 EXIT_CONNECT
Can't connect to destination (source) HN.
6 EXIT_COPY
Container private area copying/moving failed.
7 EXIT_VE_START
Can't start or restore destination CT.
8 EXIT_VE_STOP
Can't stop or checkpoint source CT.
9 EXIT_EXISTS
Container already exists on destination HN.
10 EXIT_NOTEXIST
Container does not exists on source HN.
12 EXIT_IP_INUSE
You attempt to migrate CT which IP address(es) are already in use on the destination node.
13 EXIT_QUOTA
Operation with CT quota failed.
SEE ALSO rsync(1).
COPYRIGHT
Copyright (C) 2001-2010, Parallels, Inc. Licensed under GNU GPL.
OpenVZ 28 Jun 2011 vzmigrate(8)
08-06-2012
