Sponsored Content
Top Forums Shell Programming and Scripting scp or rsync multiple files in parallel from a remote host Post 302682761 by Chubler_XL on Monday 6th of August 2012 11:25:00 PM
Old 08-07-2012
I find it hard to believe that even 1 rsync will not swamp your bandwidth as disk I/O should be WAY faster than the network.

If you want more speed try the -z (or --compress) rsync option and compress the data transfered.

Funny thing is I usually find that rsync swamps my network and even web service requests from/to the boxes concerned tend to timeout a lot. More often than not, I find myself using --bwlimit to slow things down!
This User Gave Thanks to Chubler_XL For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Backing up files to a remote host question.

I need to know how to successfully back up(and compress) files from a local machine to a remote host. Will this work? tar -cvf backup.tar -C /user/somedir | gzip backup.tar | rsh some.domain.com/user/somedir thanks in advance! (3 Replies)
Discussion started by: WeNdeL
3 Replies

2. Solaris

How to delete the files from local host to remote host

Hi all, i am copying .gz files from production server to development server using "scp" command.my requirement is after copying .gz files i want to delete old .gz files(two days back) in development server from production server. like this way i need to delelte .log ,.z and .dmp files... (3 Replies)
Discussion started by: krishna176
3 Replies

3. Shell Programming and Scripting

rsync multiple remote directories one connection

Hi, I want to specify multiple remote directories but want to transfer them in a single command with one connection with remote server. e.g. rsync -vrt --size-only --delete user@host:/home/user1/dir1 user@host:/home/user1/dir2 user@host:/home/user1/dir3 .... local_dir/ I want to... (0 Replies)
Discussion started by: sardare
0 Replies

4. Shell Programming and Scripting

rsync multiple directories into single directory on remote

I would like to find and backup all *.mp4 files from /Pictures and its sub-directories and move them to a single directory on a remote. I can find and move the files but I don't want the directory structure...just the files to be placed in a single remote directory. To find my files I use ... (1 Reply)
Discussion started by: wreckedred
1 Replies

5. Shell Programming and Scripting

scp files that are 3 days older from remote server-

hello, i am trying to get a list of files to be scped from the remote server by running the below in my local unix server ( note - there is a passwordless connectivity setup made between the local and remote server) and, we use KSH. --- ssh $scp_host "find /a/b/c/*/ -iname "$remote_file"" >... (4 Replies)
Discussion started by: billpeter3010
4 Replies

6. Shell Programming and Scripting

Pause processes in remote host and resume execution in another remote host

Hi, Given addresses of 2 remote machines, using a shell script is it possible to get the state of running processes in "src" stop all the processes in "src" exit out of "src" ssh into "dest" resume the state of executing processes captured in step 1 in "dest" Assumption: "src" is... (3 Replies)
Discussion started by: Saeya Darsan
3 Replies

7. Shell Programming and Scripting

How can we overcome Broken pipe error during scp,SFTP,Rsync while transferring big files.?

Hello All, Hope all are doing well. We use scp (some times sftp and rsync also) for transferring big files (around 2GB each ) from 1 Network to another Network. The Issues which we face :- During transfer some times( Once in 1 week (or twice)) , the speed of transfer gets down to 30 kb/s,... (2 Replies)
Discussion started by: Upendra Bhushan
2 Replies

8. UNIX for Beginners Questions & Answers

scp of multiple files to remote server

Hello, I would like to scp multiple files to a remote server that requires a password for the process to be completed. I have 30 folders (x_1, x_2 ... x_30), each containing 25 files. What I want to do is scp 1 out of the 25 files (file called bvals) for all my folders to a remote server and... (3 Replies)
Discussion started by: nasia.m
3 Replies

9. UNIX for Beginners Questions & Answers

How to paste multiple files in parallel?

Hi all, I am trying to paste thousands of files together into a matrix. Each file has only 1 column and all the files have the same number of rows (~27k rows). I tried paste * > output as well as some other for loops but the output only contains the columns from the 1st and last files. The... (8 Replies)
Discussion started by: notimenocall
8 Replies

10. Shell Programming and Scripting

Copy local files to single remote host but multiple folders using rsync

I'm trying to copy a file myfile.scr from my local Linux server to multiple folders on remote AiX server using single rsync command. Below command helps me copy the file "myfile.scr" from my localhost to a remote host folder "/app/deployment/tmpfiles" rsync --delay-updates -F --compress... (1 Reply)
Discussion started by: mohtashims
1 Replies
RoPkg::Rsync::LogParser(3pm)				User Contributed Perl Documentation			      RoPkg::Rsync::LogParser(3pm)

NAME
RoPkg::Rsync::LogParser - a rsync log parser class SYNOPSIS
#!/usr/bin/perl use strict; use warning; sub main { my $lp = new RoPkg::Rsync::LogParser( type => 'client' ); $lp->Parse('/tmp/debian-log.txt'); } main(); Short description RoPkg::Rsync::LogParser is a class used to parse (for the moment) client side rsync logs. Using this class you can extract the transfered files, symlinks created, directories, total number of files, transfered bytes, speed of transfer etc. LogParser support list filter trough probes (used by many projects to verify the mirrors). The primary use of this class is inside Simba, but can be used without problems in any other project who needs a rsync client log parser. METHODS
new(hash) Constructor of the class. When creating a new class instance you must provide the type of log: client or server . For the moment only the client parsing routines are defined. Log($raw_log) Get/Set the raw rsync log. With no parameters, the get behaviour is selected. If a parameter is present (and valid) the method acts as a Set method. Probes(@probes_list) Get/Set the raw probes list. If no parameter is present, the method returns the number of probes defined (0 if none). If a list of probes is given, the list is added to the existing list of probes. Parse($filename) Parse a rsync client log. If no filename is specified, the method parses the log previsiously set with Log() method. Please take extra care with the filename. If it does not exists, or is not a file a exception will be raised (Param::Wrong and File::Open). Files() In scalar context returns the number of transfered files (filtered by probes list). In list context, returns the list of transfered files. RealFiles() In scalar context returns the number of transfered files. In list context, returns the list of transfered files. Deleted() In scalar context returns the number of deleted files (filtered by probes list). In list context, returns the list of deleted files. RealDeleted() In scalar context returns the number of deleted files. In list context, returns the list of deleted files. Speed() Returns the speed (as reported by rsync [in bytes/sec]) of the transfer. Size() Returns the size of the synced content. TransfData() Returns the total number of bytes transfered during the sync session Symlinks() In scalar context returns the number of symlinks (filtered by probes list). In list context, returns the list of symlinks. RealSymlinks() In scalar context returns the number of symlinks. In list context, returns the list of symlinks. PREREQUISITES
RoPkg::Utils requires perl 5.008 or later and RoPkg::Exceptions class. From CPAN, you need the Scalar::Util module. At build time Test::More is required for tests (if you don't plan to use the tests ignore this dependency). SEE ALSO
RoPkg::Rsync::Node RoPkg::Rsync::ConfFile RoPkg::Exceptions AUTHOR
Subredu Manuel <diablo@iasi.roedu.net> LICENSE
Copyright (C) 2005 Subredu Manuel. All Rights Reserved. This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. The LICENSE file contains the full text of the license. perl v5.8.8 2006-06-09 RoPkg::Rsync::LogParser(3pm)
All times are GMT -4. The time now is 05:05 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy