10-08-2019
It could be some sort of resource issue, or connection limit. I'd try less threads.
With disk and network I/O bottlenecks involved, I find it hard to believe that 40 threads would be quicker than 5, on my low spec system here 5 takes about twice as long as 1!
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi
I need to move multiple (say 10 files) from one location to another location. My selection would be like this...
ls -ltr *.arc | head ---> Need to move top 10 files with single command without iterating in loop. I know we can move files like this with find command but not sure if I can... (4 Replies)
Discussion started by: malaymaru
4 Replies
2. Shell Programming and Scripting
I believe what is happening is rm is executing in the script on every directory and on failure of the first it stops although returns status 0.
find $HOME -name /directory/filename | xargs -l rm
This is the code I use but file remains. I am using sun solaris system which has way limited... (4 Replies)
Discussion started by: Ebodee
4 Replies
3. Shell Programming and Scripting
Hi,
I have many files like below in my currect dir
test1.me
test2.me
test3.me
I wanted them to rename without .me extention
I tried below
find . -name "*.me" | xargs -i mv {} `echo {} | sed 's/\.me//'`
It thorws error that
cant mv ./test1.me to ./test1.me as they are... (6 Replies)
Discussion started by: sunilmenhdiratt
6 Replies
4. Shell Programming and Scripting
I need to apply mp3gain (album mode) to all mp3 files in a given directory. Each album is in its own directory under /media/data/music/albums for example:
/media/data/music/albums/foo
/media/data/music/albums/bar
/media/data/music/albums/more
What needs to happen is:
cd... (4 Replies)
Discussion started by: audiophile
4 Replies
5. Shell Programming and Scripting
Hi, I'd like to ask if anybody can help improve my code to move 1 million+ files from a directory to another:
find /source/dir -name file* -type f | xargs -I '{}' mv {} /destination/dir
I learned this line of code from this forum as well and it works fine. However, file movement is kinda... (6 Replies)
Discussion started by: agentgrecko
6 Replies
6. Shell Programming and Scripting
Hello Experts,
I need to copy files from one location to another using xargs.
Tried something like this (In Ubuntu & Solaris ).
mkdir -p 1234;
find /home/emd/Desktop/n007/M007/ -type f -name "A2014*" | xargs -0 cp -r {} /home/emd/Desktop/1234
But every time i run this, a weird error... (6 Replies)
Discussion started by: Saidul
6 Replies
7. Shell Programming and Scripting
Hello,
i have an dynamical apache_cache that I need to clean all days (older tant one day) with an unix command :
find /usr/IBM/HTTPServer/apache_cache/ -name '*' -mtime +1 -print0|xargs -0 rm -r --
but it didn't work.
Could you explain me why.
So I will put below all my script :... (13 Replies)
Discussion started by: steiner
13 Replies
8. Shell Programming and Scripting
I want to run parallelise rsync with xargs. The plan is to
take separate directories and run rsync in parallel.
find . -mindepth 1 -maxdepth 1 print0 | xargs -n1 -P0 rsync -avz
Hoxever I am stuck on how to use rsync in this way
---------- Post updated at 04:44 AM ---------- Previous... (3 Replies)
Discussion started by: kristinu
3 Replies
9. Shell Programming and Scripting
Please, i have a question about rsync command:
Here is the command that i have used in my script:
rsync -ratlz --rsh="/usr/bin/sshpass ssh -o StrictHostKeyChecking=no" -aAXHltzh --progress --numeric-ids --devices --rsync-path="rsync" $1 $hostname:$1
using this command, i can... (0 Replies)
Discussion started by: chercheur111
0 Replies
10. Shell Programming and Scripting
I am running the below loop that to process the 3 bam files (which isn't always the case). A .py executable is then called using | xargs sh to further process. If I just run it with echo the output is fine and expected, however when
| xargs sh is added I get the error. I tried adding | xargs... (4 Replies)
Discussion started by: cmccabe
4 Replies
LEARN ABOUT DEBIAN
parallel-rsync
PARALLEL-RSYNC(1) PARALLEL-RSYNC(1)
NAME
parallel-rsync - deploy files to listed hosts
SYNOPSIS
parallel-rsync [OPTIONS] -h hosts.txt local remote
DESCRIPTION
pssh provides a number of commands for executing against a group of computers, using SSH. It's most useful for operating on clusters of
homogenously-configured hosts.
parallel-rsync deploy files files to all hosts you listed.
OPTIONS
-r --recursive
recusively copy directories (OPTIONAL)
-a ----archive
use rsync -a (archive mode) (OPTIONAL)
-z --compress
use rsync compression (OPTIONAL)
-h --hosts
hosts file (each line "host[:port] [user]")
-l --user
username (OPTIONAL)
-p --par
max number of parallel threads (OPTIONAL)
-o --outdir
output directory for stdout files (OPTIONAL)
-e --errdir
output directory for stderr files (OPTIONAL)
-t --timeout
timeout (secs) (-1 = no timeout) per host (OPTIONAL)
-O --options
SSH options (OPTIONAL)
-v --verbose
turn on warning and diagnostic messages (OPTIONAL)
EXAMPLE
# parallel-rsync -r -h hosts.txt -l irb2 foo /home/irb2/foo
ENVIRONMENT
All four programs take similar sets of options. All of these options can be set using the following environment variables:
o PSSH_HOSTS
o PSSH_USER
o PSSH_PAR
o PSSH_OUTDIR
o PSSH_VERBOSE
o PSSH_OPTIONS
SEE ALSO
parallel-ssh(1), parallel-scp(1), parallel-slurp(1), parallel-nuke(1), ssh(1), rsync(1)
AUTHOR
Brent N. Chun <bnc@theether.org>
COPYING
Copyright: 2003, 2004, 2005, 2006, 2007 Brent N. Chun
NOTES
1. bnc@theether.org
mailto:bnc@theether.org
03/30/2009 PARALLEL-RSYNC(1)