scp or rsync multiple files in parallel from a remote host
Hi.
I'm trying to speed up an rsync command by running it in parallel. There's no real option for this other than if the files are in multiple directories (which they're not). And even then there's no way of knowing if rsync has succeeded as the process is running in the background .. and there's no return code from the process id. Any ideas?
I thought about just using scp but scripting it is proving trickier than I thought and I'm getting errors on the for loop when I try to introduce variables into the ssh part:
E.g.
Code:
RHOST=srv245005
CCMD="ls /db/remote_backup_area"
for BackupFile in 'ssh -q oracle@${RHOST} ${CCMD}'; do
echo "attempting to copy file: " $BackupFile
scp -p $hostname:/db/remote_backup_area/$file /db/local_dump_area
# or use rsync
#rsync -av -e ssh oracle@${RHOST}:/db/remote_backup_area/${BackupFile} /db/local_dump_area"
done
exit
Any help much appreciated.
Moderator's Comments:
code tags please
Last edited by jim mcnamara; 08-06-2012 at 02:47 PM..
I need to know how to successfully back up(and compress) files from a local machine to a remote host.
Will this work?
tar -cvf backup.tar -C /user/somedir | gzip backup.tar | rsh some.domain.com/user/somedir
thanks in advance! (3 Replies)
Hi all,
i am copying .gz files from production server to development server using
"scp" command.my requirement is after copying .gz files i want to delete old
.gz files(two days back) in development server from production server.
like this way i need to delelte .log ,.z and .dmp files... (3 Replies)
Hi,
I want to specify multiple remote directories but want to transfer them in a single command with one connection with remote server.
e.g.
rsync -vrt --size-only --delete user@host:/home/user1/dir1 user@host:/home/user1/dir2 user@host:/home/user1/dir3 .... local_dir/
I want to... (0 Replies)
I would like to find and backup all *.mp4 files from /Pictures and its sub-directories and move them to a single directory on a remote. I can find and move the files but I don't want the directory structure...just the files to be placed in a single remote directory.
To find my files I use
... (1 Reply)
hello,
i am trying to get a list of files to be scped from the remote server
by running the below in my local unix server ( note - there is a passwordless connectivity setup made between the local and remote server) and, we use KSH.
---
ssh $scp_host "find /a/b/c/*/ -iname "$remote_file"" >... (4 Replies)
Hi,
Given addresses of 2 remote machines, using a shell script is it possible to
get the state of running processes in "src"
stop all the processes in "src"
exit out of "src"
ssh into "dest"
resume the state of executing processes captured in step 1 in "dest"
Assumption:
"src" is... (3 Replies)
Hello All,
Hope all are doing well.
We use scp (some times sftp and rsync also) for transferring big files (around 2GB each ) from 1 Network to another Network.
The Issues which we face :-
During transfer some times( Once in 1 week (or twice)) , the speed of transfer gets down to 30 kb/s,... (2 Replies)
Hello,
I would like to scp multiple files to a remote server that requires a password for the process to be completed.
I have 30 folders (x_1, x_2 ... x_30), each containing 25 files. What I want to do is scp 1 out of the 25 files (file called bvals) for all my folders to a remote server and... (3 Replies)
Hi all,
I am trying to paste thousands of files together into a matrix. Each file has only 1 column and all the files have the same number of rows (~27k rows). I tried
paste * > output as well as some other for loops
but the output only contains the columns from the 1st and last files. The... (8 Replies)
I'm trying to copy a file myfile.scr from my local Linux server to multiple folders on remote AiX server using single rsync command.
Below command helps me copy the file "myfile.scr" from my localhost to a remote host folder "/app/deployment/tmpfiles"
rsync --delay-updates -F --compress... (1 Reply)
Discussion started by: mohtashims
1 Replies
LEARN ABOUT DEBIAN
dput.cf
dput.cf(5) File Formats Manual dput.cf(5)NAME
dput.cf - Debian package upload tool configuration file
DESCRIPTION
This manpage gives a brief overview of dput's configuration file and the available options in it. dput is a tool to upload Debian packages
to the archive.
FORMAT
dput.cf consists of different groups of configuration options, one for each host where you want to be able to upload packages. Hosts are
defined using an identifier header with a short name for the host, enclosed in square brackets. Note that only if multiple such headers
are encountered in the configuration, only the group following the last header is considered. This is done to avoid confusion when overrid-
ing a global configuration file with a user-specific one. There's a special identifier, [DEFAULT], which holds default parameters for all
the hosts. The defaults can be overridden by redefining them again in each host section. The available parameters are listed below:
fqdn This is the fully qualified domain name that will be used (can be specified as host:port for HTTP, HTTPS and FTP).
login Your login on the machine named before. A single asterisk * will cause the scp and rsync uploaders to not use supply a login name
when calling to ssh, scp, and rsync.
incoming
The directory that you should upload the files to.
method The method that you want to use for uploading the files. Currently, dput accepts the following values for method:
ftp the package will be uploaded via ftp, either anonymously or using a login/password. Note that ftp is unencrypted so you
should not use password authentication with this.
http and https
the package will be uploaded via http or https using the PUT method as specified in WebDAV. The upload method will prompt for
a password if necessary.
scp the package will be uploaded using ssh's scp. This transfers files using a secure ssh tunnel, and needs an account on the
upload machine.
rsync the package will be uploaded using rsync over ssh. This is similar to scp, but can save some bandwidth if the destination
file already exists on the upload server. It also needs a login on the remote machine as it uses ssh.
local the package will be "uploaded" locally using /usr/bin/install. This transfers files to a local incoming directory, and needs
appropriate permissions set on that directory.
hash The hash algorithm that should be used in calculating the checksum of the files before uploading them. Currently, dput accepts the
following values for hash:
md5 use the md5 algorithm for calculation
sha use the sha algorithm for calculation
allow_unsigned_uploads
This defines if you are allowed to upload files without a GnuPG signature to this host or not.
allow_dcut
This defines if you are allowed to upload a dcut changes file to the queue to remove or move files.
distributions
This defines a comma-separated list of distributions that this host accepts, used to guess the host to use when none is given on the
command line.
allowed_distributions
A regular expression (of Python re module syntax) that the distribution field must match or dput will refuse the upload.
delayed
Set a numeric default parameter for delayed uploads (i.e. uploads to this queue will be delayed the specified number of days.
Defaults to the empty string, meaning no delay. This only works with upload queues that support delayed uploads.
run_lintian
This option defines if lintian should be run before the package will be uploaded or not. If the package is not lintian clean, the
upload will not happen.
run_dinstall
This options defines if dinstall -n should be run after the package has been uploaded or not. This is an easy way to test if your
package would be installed into the archive or not.
check_version
This option defines if dput should check if the user has installed the package in his system for testing it before putting it into
the archive. If the user has not installed and tested it, dput will reject the upload.
passive_ftp
This option defines if dput should use passive ftp or active ftp for uploading a package to one of the upload queues. By default,
dput uses passive ftp connections. If you need to use active ftp connections, set passive_ftp to 0.
progress_indicator
This integer option defines if dput should display a progress indicator for the upload. (Currently implemented in ftp only.)
Supported values: 0 (default) - no progress, 1 - rotating progress indicator, and 2 - kilobyte counter.
scp_compress
This option defines if the scp upload to the host will be compressed, or not. This option is only used for the 'scp' upload method,
and has been found to decrease upload time for slow links, and increase upload times for faster links.
ssh_config_options
The arguments of this config options should be ssh config file options in the style documented in ssh_config(5). They will be passed
to all automatic invocations of ssh and scp by dput. Note that you can define multiline (dput) configuration options by indenting
the second line with whitespace (i.e. similar to RFC822 header continuations).
post_upload_command
This option defines a command to be run by dput after a successful upload.
pre_upload_command
This option defines a command to be run by dput before a upload happens.
default_host_main
This defines the default host for packages that are allowed to be uploaded to the main archive. This variable is used when guessing
the host to upload to.
BUGS
Please send bug reports to the author.
FILES
/etc/dput.cf
global dput configuration file
~/.dput.cf
peruser dput configuration file
AUTHOR
Christian Kurz. Updated by Thomas Viehmann <tv@beamnet.de>.
Many other people have contributed to this code. See the Thanks file.
SEE ALSO dput(1)
/usr/share/doc/dput
COMMENTS
The author appreciates comments and suggestions from you, if any.
April 8, 2001 dput.cf(5)
08-06-2012
