Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Xargs and rsync not sending all files
Top Forums Shell Programming and Scripting Xargs and rsync not sending all files Post 303039488 by Chubler_XL on Monday 7th of October 2019 06:29:45 PM
Old 10-07-2019
This seems to be working OK for me. I changed script as follows, so I could test in on local files:

Code:
#!/bin/bash
  
# SETUP OPTIONS
export SRCDIR="./test/sapdata*"
export FSRCDIR="./scratch/doug/test"
export DESTDIR="TST"
export THREADS="40"

echo "Starting... "
date

# RSYNC DIRECTORY STRUCTURE
rsync -zvr -f"- *sapdata*/.snapshot" -f"+ */" -f"- *" ${SRCDIR} ${DESTDIR} \

# FIND ALL FILES AND PASS THEM TO MULTIPLE RSYNC PROCESSES
cd $FSRCDIR  &&  find sapdata* -path "sapdata*/.snapshot" -prune -o ! -type d -print0 | xargs -0 -n1 -P${THREADS} -I% rsync -arqh --partial --size-only" % ../../../${DESTDIR}/%

echo "Complete"
date

And tested it like this:

Code:
$  mkdir -p TST {test,scratch/doug/test}/sapdata{1..6}/dir{1..5}
$ touch scratch/doug/test/sapdata{1..6}/dir{1..5}/file{a..z}
$ ./dougCopy
Starting... 
Tue, Oct 08, 2019  8:21:17 AM
sending incremental file list
sapdata1/
sasdata1/dir1/
...
sapdata6/dir5/
sent 774 bytes  received 156 bytes  1,860.00 bytes/sec
total size is 0  speedup is 0.00
Complete
Tue, Oct 08, 2019  8:22:15 AM
$  find scratch/doug/test/ -type f -print | wc -l
780
$ find TST -type f -print | wc -l
780

This User Gave Thanks to Chubler_XL For This Post:
DougyC
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

MV files with xargs or -exec

Hi I need to move multiple (say 10 files) from one location to another location. My selection would be like this... ls -ltr *.arc | head ---> Need to move top 10 files with single command without iterating in loop. I know we can move files like this with find command but not sure if I can... (4 Replies)
Discussion started by: malaymaru
4 Replies

2. Shell Programming and Scripting

find with xargs to rm found files

I believe what is happening is rm is executing in the script on every directory and on failure of the first it stops although returns status 0. find $HOME -name /directory/filename | xargs -l rm This is the code I use but file remains. I am using sun solaris system which has way limited... (4 Replies)
Discussion started by: Ebodee
4 Replies

3. Shell Programming and Scripting

move files using xargs

Hi, I have many files like below in my currect dir test1.me test2.me test3.me I wanted them to rename without .me extention I tried below find . -name "*.me" | xargs -i mv {} `echo {} | sed 's/\.me//'` It thorws error that cant mv ./test1.me to ./test1.me as they are... (6 Replies)
Discussion started by: sunilmenhdiratt
6 Replies

4. Shell Programming and Scripting

help using find/xargs to apply mp3gain to files

I need to apply mp3gain (album mode) to all mp3 files in a given directory. Each album is in its own directory under /media/data/music/albums for example: /media/data/music/albums/foo /media/data/music/albums/bar /media/data/music/albums/more What needs to happen is: cd... (4 Replies)
Discussion started by: audiophile
4 Replies

5. Shell Programming and Scripting

Fast processing(mv command) of 1 million+ files using find, mv and xargs

Hi, I'd like to ask if anybody can help improve my code to move 1 million+ files from a directory to another: find /source/dir -name file* -type f | xargs -I '{}' mv {} /destination/dir I learned this line of code from this forum as well and it works fine. However, file movement is kinda... (6 Replies)
Discussion started by: agentgrecko
6 Replies

6. Shell Programming and Scripting

How to copy files from one location to another using xargs??

Hello Experts, I need to copy files from one location to another using xargs. Tried something like this (In Ubuntu & Solaris ). mkdir -p 1234; find /home/emd/Desktop/n007/M007/ -type f -name "A2014*" | xargs -0 cp -r {} /home/emd/Desktop/1234 But every time i run this, a weird error... (6 Replies)
Discussion started by: Saidul
6 Replies

7. Shell Programming and Scripting

How to delete directories and files with xargs?

Hello, i have an dynamical apache_cache that I need to clean all days (older tant one day) with an unix command : find /usr/IBM/HTTPServer/apache_cache/ -name '*' -mtime +1 -print0|xargs -0 rm -r -- but it didn't work. Could you explain me why. So I will put below all my script :... (13 Replies)
Discussion started by: steiner
13 Replies

8. Shell Programming and Scripting

Using rsync with xargs

I want to run parallelise rsync with xargs. The plan is to take separate directories and run rsync in parallel. find . -mindepth 1 -maxdepth 1 print0 | xargs -n1 -P0 rsync -avz Hoxever I am stuck on how to use rsync in this way ---------- Post updated at 04:44 AM ---------- Previous... (3 Replies)
Discussion started by: kristinu
3 Replies

9. Shell Programming and Scripting

Linux command rsync sending files incrementally

Please, i have a question about rsync command: Here is the command that i have used in my script: rsync -ratlz --rsh="/usr/bin/sshpass ssh -o StrictHostKeyChecking=no" -aAXHltzh --progress --numeric-ids --devices --rsync-path="rsync" $1 $hostname:$1 using this command, i can... (0 Replies)
Discussion started by: chercheur111
0 Replies

10. Shell Programming and Scripting

Xargs to call python executable to process multiple bam files

I am running the below loop that to process the 3 bam files (which isn't always the case). A .py executable is then called using | xargs sh to further process. If I just run it with echo the output is fine and expected, however when | xargs sh is added I get the error. I tried adding | xargs... (4 Replies)
Discussion started by: cmccabe
4 Replies

LEARN ABOUT DEBIAN

colorgccrc

COLORGCCRC(5)							File Formats Manual						     COLORGCCRC(5)

NAME

       colorgccrc - configuration file for colorgcc

DESCRIPTION

       A colorgccrc configuration file is used to configure the highlighting of the compiler output from colorgcc.

SYNTAX

       Each  line  consists  of  a  keyword  designating  a configuration variable.  The keyword is followed by `:' and then one or several values
       (depending on the keyword).  Lines beginning with a hash mark `#' are comments.

CONFIGURATION VARIABLES

       g++ | gcc | c++ | cc | g77 | gcj | gnat | gpc
	      Specifies the paths to the compilers.  Takes one value; a path to the compiler.

       nocolor
	      Specifies what terminal types colorization should be disabled on.  Takes one or several values, separated by whitespace.

       srcColor
	      Specifies the highlighting attributes source-code should be given.  Takes one or several color attributes.  See  the  section  COLOR
	      ATTRIBUTES for more information.

       introColor
	      Specifies  the  highlighting  attributes	for normal compiler output.  Takes one or several color attributes.  See the section COLOR
	      ATTRIBUTES for more information.

       warningFileNameColor | errorFileNameColor
	      Specifies the highlighting attributes for the filename in a  warning  or	an  error,  respectively.   Takes  one	or  several  color
	      attributes.  See the section COLOR ATTRIBUTES for more information.

       warningNumberColor | errorNumberColor
	      Specifies  the  highlighting  attributes	for  the  line-number  in a warning or an error, respectively.	Takes one or several color
	      attributes.  See the section COLOR ATTRIBUTES for more information.

       warningMessageColor | errorMessageColor
	      Specifies the highlighting attributes for the message-text in a warning or an error,  respectively.   Takes  one	or  several  color
	      attributes.  See the section COLOR ATTRIBUTES for more information.

COLOR ATTRIBUTES

       The following attributes are valid for highlighting.

       clear, reset

       bold, underline, underscore, blink, reverse, concealed

       black, red, green, yellow, blue, magenta, cyan, white

       on_black, on_red, on_green, on_yellow, on_blue, on_magenta, on_cyan, on_white

SEE ALSO

       gcc(1), colorgcc(1)

HISTORY

       Jan 15 2003: Initial version of this manual-page.

REPORTING BUGS

       Report bugs to <jmoyers@geeks.com>

AUTHORS

       Jamie Moyers <jmoyers@geeks.com> is the author of colorgcc.

       This  manual  page  was	written  by Joe Wreschnig <piman@sacredchao.net>, and modified by David Weinehall <tao@debian.org>, for the Debian
       GNU/Linux system (but may be used by others).

COPYRIGHT

       Copyright (C) 2003 Jamie Moyers
       This is free software; see the source for copying conditions.  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICU-
       LAR PURPOSE.

								   Jan 15, 2003 						     COLORGCCRC(5)
All times are GMT -4. The time now is 07:10 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy