Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Xargs and rsync not sending all files
Top Forums Shell Programming and Scripting Xargs and rsync not sending all files Post 303039476 by DougyC on Monday 7th of October 2019 09:31:05 AM
Old 10-07-2019
[SOLVED} Xargs and rsync not sending all files
Hi all,

I have a script issue I can't seem to work out.

In a directory I have several folders and I want to send just the sapdata1 to sapdata14 folders and their contents but not sapdataXX/.snapshot

the script is:
Code:
#!/bin/bash

# SETUP OPTIONS
export SRCDIR="/scratch/doug/test/sapdata*"
export FSRCDIR="/scratch/doug/test"
export DESTDIR="fred@111.222.333.444:/sendtest/TST"
export THREADS="40"

echo "Starting... "
date

# RSYNC DIRECTORY STRUCTURE
rsync -zvr -f"- *sapdata*/.snapshot" -f"+ */" -f"- *" ${SRCDIR} ${DESTDIR} \

# FIND ALL FILES AND PASS THEM TO MULTIPLE RSYNC PROCESSES
cd $FSRCDIR  &&  find sapdata* -path "sapdata*/.snapshot" -prune -o ! -type d -print0 | xargs -0 -n1 -P${THREADS} -I% rsync -arvh --partial --size-only % ${DESTDIR}/%

echo "Complete"
date

The first rsync seems to work, I can see all the folders at the destination.

The find works and finds just over 600 files, however the xargs command only seems to send 40 files and then stops, I thought the -P would spawn 40 processes at a time until all the files were sent . I'm probably missing something obvious but I just can't figure it out.

I should also mention that I'm sending about 15Tb worth of large files. Hence the send in parallel script.

Any help appreciated.
Thanks
Last edited by DougyC; 10-09-2019 at 05:50 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

MV files with xargs or -exec

Hi I need to move multiple (say 10 files) from one location to another location. My selection would be like this... ls -ltr *.arc | head ---> Need to move top 10 files with single command without iterating in loop. I know we can move files like this with find command but not sure if I can... (4 Replies)
Discussion started by: malaymaru
4 Replies

2. Shell Programming and Scripting

find with xargs to rm found files

I believe what is happening is rm is executing in the script on every directory and on failure of the first it stops although returns status 0. find $HOME -name /directory/filename | xargs -l rm This is the code I use but file remains. I am using sun solaris system which has way limited... (4 Replies)
Discussion started by: Ebodee
4 Replies

3. Shell Programming and Scripting

move files using xargs

Hi, I have many files like below in my currect dir test1.me test2.me test3.me I wanted them to rename without .me extention I tried below find . -name "*.me" | xargs -i mv {} `echo {} | sed 's/\.me//'` It thorws error that cant mv ./test1.me to ./test1.me as they are... (6 Replies)
Discussion started by: sunilmenhdiratt
6 Replies

4. Shell Programming and Scripting

help using find/xargs to apply mp3gain to files

I need to apply mp3gain (album mode) to all mp3 files in a given directory. Each album is in its own directory under /media/data/music/albums for example: /media/data/music/albums/foo /media/data/music/albums/bar /media/data/music/albums/more What needs to happen is: cd... (4 Replies)
Discussion started by: audiophile
4 Replies

5. Shell Programming and Scripting

Fast processing(mv command) of 1 million+ files using find, mv and xargs

Hi, I'd like to ask if anybody can help improve my code to move 1 million+ files from a directory to another: find /source/dir -name file* -type f | xargs -I '{}' mv {} /destination/dir I learned this line of code from this forum as well and it works fine. However, file movement is kinda... (6 Replies)
Discussion started by: agentgrecko
6 Replies

6. Shell Programming and Scripting

How to copy files from one location to another using xargs??

Hello Experts, I need to copy files from one location to another using xargs. Tried something like this (In Ubuntu & Solaris ). mkdir -p 1234; find /home/emd/Desktop/n007/M007/ -type f -name "A2014*" | xargs -0 cp -r {} /home/emd/Desktop/1234 But every time i run this, a weird error... (6 Replies)
Discussion started by: Saidul
6 Replies

7. Shell Programming and Scripting

How to delete directories and files with xargs?

Hello, i have an dynamical apache_cache that I need to clean all days (older tant one day) with an unix command : find /usr/IBM/HTTPServer/apache_cache/ -name '*' -mtime +1 -print0|xargs -0 rm -r -- but it didn't work. Could you explain me why. So I will put below all my script :... (13 Replies)
Discussion started by: steiner
13 Replies

8. Shell Programming and Scripting

Using rsync with xargs

I want to run parallelise rsync with xargs. The plan is to take separate directories and run rsync in parallel. find . -mindepth 1 -maxdepth 1 print0 | xargs -n1 -P0 rsync -avz Hoxever I am stuck on how to use rsync in this way ---------- Post updated at 04:44 AM ---------- Previous... (3 Replies)
Discussion started by: kristinu
3 Replies

9. Shell Programming and Scripting

Linux command rsync sending files incrementally

Please, i have a question about rsync command: Here is the command that i have used in my script: rsync -ratlz --rsh="/usr/bin/sshpass ssh -o StrictHostKeyChecking=no" -aAXHltzh --progress --numeric-ids --devices --rsync-path="rsync" $1 $hostname:$1 using this command, i can... (0 Replies)
Discussion started by: chercheur111
0 Replies

10. Shell Programming and Scripting

Xargs to call python executable to process multiple bam files

I am running the below loop that to process the 3 bam files (which isn't always the case). A .py executable is then called using | xargs sh to further process. If I just run it with echo the output is fine and expected, however when | xargs sh is added I get the error. I tried adding | xargs... (4 Replies)
Discussion started by: cmccabe
4 Replies

LEARN ABOUT SUNOS

evolution-addressbook-export

evolution-addressbook-export(1) 				   User Commands				   evolution-addressbook-export(1)

NAME

       evolution-addressbook-export - export local address book from Ximian Evolution, Sun Microsystems Edition

SYNOPSIS

       evolution-addressbook-export  [--async] [--format=[vcard|csv]] [--help] [--list-addressbook-folders] [--output=output-file] [--size=number]
       [--usage] [--version]

DESCRIPTION

       Export the local address book to a file.

OPTIONS

       The following options are supported:

       --async		       Export in asynchronous mode. You can also use -a to specify this option.

       --format=[vcard|csv]    Specify the export format:  virtual card (vcard) or comma-separated value (csv).

       --help		       Display help text. You can also use -? to specify this option.

       --list-addressbook-foldeList the local address-book folders. You can also use -l to specify this option.

       --output=output-file    Specify the name of the output file to store the exported information. If no output file is specified, the exported
			       information is sent to standard output.

       --size=number	       Specify the number of cards in one output file in asychronous mode. The default value is 100.

       --usage		       Display brief usage text.

       --version	       Display version information.

EXAMPLES

       Example 1: Exporting Local Address Book in Asynchronous Mode and Vcard Format to contacts.vcf

       example% evolution-addressbook-export -a --format=vcard --output=contacts.vcf

EXIT STATUS

       The following exit values are returned:

       0	Application exited successfully

       1	Application exited with failure

FILES

       The following files are used by this application:

       /usr/bin/evolution-addressbook-export

	   Executable to export Evolution local address book

       $HOME/evolution

	   Per-user configuration files and local storage for Evolution

       $HOME/evolution/mail/account/folders

	   Temporary files for Evolution mail

       $HOME/evolution/sunone/account/folders

	   Temporary files for Evolution Sun ONE account

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Architecture		     |SPARC			   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWevo			   |
       +-----------------------------+-----------------------------+
       |CSI			     |Enabled			   |
       +-----------------------------+-----------------------------+
       |Interface stability	     |Standard			   |
       +-----------------------------+-----------------------------+
       |MT-Level		     |MT safe with Exceptions	   |
       +-----------------------------+-----------------------------+

SEE ALSO

       Ximian Evolution Sun Microsystems Edition User Guide

       evolution(1), evolution-address-import(1)

NOTES

       Written by Steven Zhang, Sun Microsystems Inc., 2004.

SunOS 5.10							    15 Apr 2004 				   evolution-addressbook-export(1)
All times are GMT -4. The time now is 05:28 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy