Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Xargs and rsync not sending all files
Top Forums Shell Programming and Scripting Xargs and rsync not sending all files Post 303039476 by DougyC on Monday 7th of October 2019 09:31:05 AM
Old 10-07-2019
[SOLVED} Xargs and rsync not sending all files
Hi all,

I have a script issue I can't seem to work out.

In a directory I have several folders and I want to send just the sapdata1 to sapdata14 folders and their contents but not sapdataXX/.snapshot

the script is:
Code:
#!/bin/bash

# SETUP OPTIONS
export SRCDIR="/scratch/doug/test/sapdata*"
export FSRCDIR="/scratch/doug/test"
export DESTDIR="fred@111.222.333.444:/sendtest/TST"
export THREADS="40"

echo "Starting... "
date

# RSYNC DIRECTORY STRUCTURE
rsync -zvr -f"- *sapdata*/.snapshot" -f"+ */" -f"- *" ${SRCDIR} ${DESTDIR} \

# FIND ALL FILES AND PASS THEM TO MULTIPLE RSYNC PROCESSES
cd $FSRCDIR  &&  find sapdata* -path "sapdata*/.snapshot" -prune -o ! -type d -print0 | xargs -0 -n1 -P${THREADS} -I% rsync -arvh --partial --size-only % ${DESTDIR}/%

echo "Complete"
date

The first rsync seems to work, I can see all the folders at the destination.

The find works and finds just over 600 files, however the xargs command only seems to send 40 files and then stops, I thought the -P would spawn 40 processes at a time until all the files were sent . I'm probably missing something obvious but I just can't figure it out.

I should also mention that I'm sending about 15Tb worth of large files. Hence the send in parallel script.

Any help appreciated.
Thanks
Last edited by DougyC; 10-09-2019 at 05:50 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

MV files with xargs or -exec

Hi I need to move multiple (say 10 files) from one location to another location. My selection would be like this... ls -ltr *.arc | head ---> Need to move top 10 files with single command without iterating in loop. I know we can move files like this with find command but not sure if I can... (4 Replies)
Discussion started by: malaymaru
4 Replies

2. Shell Programming and Scripting

find with xargs to rm found files

I believe what is happening is rm is executing in the script on every directory and on failure of the first it stops although returns status 0. find $HOME -name /directory/filename | xargs -l rm This is the code I use but file remains. I am using sun solaris system which has way limited... (4 Replies)
Discussion started by: Ebodee
4 Replies

3. Shell Programming and Scripting

move files using xargs

Hi, I have many files like below in my currect dir test1.me test2.me test3.me I wanted them to rename without .me extention I tried below find . -name "*.me" | xargs -i mv {} `echo {} | sed 's/\.me//'` It thorws error that cant mv ./test1.me to ./test1.me as they are... (6 Replies)
Discussion started by: sunilmenhdiratt
6 Replies

4. Shell Programming and Scripting

help using find/xargs to apply mp3gain to files

I need to apply mp3gain (album mode) to all mp3 files in a given directory. Each album is in its own directory under /media/data/music/albums for example: /media/data/music/albums/foo /media/data/music/albums/bar /media/data/music/albums/more What needs to happen is: cd... (4 Replies)
Discussion started by: audiophile
4 Replies

5. Shell Programming and Scripting

Fast processing(mv command) of 1 million+ files using find, mv and xargs

Hi, I'd like to ask if anybody can help improve my code to move 1 million+ files from a directory to another: find /source/dir -name file* -type f | xargs -I '{}' mv {} /destination/dir I learned this line of code from this forum as well and it works fine. However, file movement is kinda... (6 Replies)
Discussion started by: agentgrecko
6 Replies

6. Shell Programming and Scripting

How to copy files from one location to another using xargs??

Hello Experts, I need to copy files from one location to another using xargs. Tried something like this (In Ubuntu & Solaris ). mkdir -p 1234; find /home/emd/Desktop/n007/M007/ -type f -name "A2014*" | xargs -0 cp -r {} /home/emd/Desktop/1234 But every time i run this, a weird error... (6 Replies)
Discussion started by: Saidul
6 Replies

7. Shell Programming and Scripting

How to delete directories and files with xargs?

Hello, i have an dynamical apache_cache that I need to clean all days (older tant one day) with an unix command : find /usr/IBM/HTTPServer/apache_cache/ -name '*' -mtime +1 -print0|xargs -0 rm -r -- but it didn't work. Could you explain me why. So I will put below all my script :... (13 Replies)
Discussion started by: steiner
13 Replies

8. Shell Programming and Scripting

Using rsync with xargs

I want to run parallelise rsync with xargs. The plan is to take separate directories and run rsync in parallel. find . -mindepth 1 -maxdepth 1 print0 | xargs -n1 -P0 rsync -avz Hoxever I am stuck on how to use rsync in this way ---------- Post updated at 04:44 AM ---------- Previous... (3 Replies)
Discussion started by: kristinu
3 Replies

9. Shell Programming and Scripting

Linux command rsync sending files incrementally

Please, i have a question about rsync command: Here is the command that i have used in my script: rsync -ratlz --rsh="/usr/bin/sshpass ssh -o StrictHostKeyChecking=no" -aAXHltzh --progress --numeric-ids --devices --rsync-path="rsync" $1 $hostname:$1 using this command, i can... (0 Replies)
Discussion started by: chercheur111
0 Replies

10. Shell Programming and Scripting

Xargs to call python executable to process multiple bam files

I am running the below loop that to process the 3 bam files (which isn't always the case). A .py executable is then called using | xargs sh to further process. If I just run it with echo the output is fine and expected, however when | xargs sh is added I get the error. I tried adding | xargs... (4 Replies)
Discussion started by: cmccabe
4 Replies

LEARN ABOUT DEBIAN

vzdump

vzdump(1)						       Proxmox Documentation							 vzdump(1)

NAME

       vzdump - backup utility for virtual machine

SYNOPSIS

       vzdump OPTIONS [--all | <VMID>]

       --exclude VMID	       exclude VMID (assumes --all)

       --exclude-path REGEX    exclude certain files/directories. You
			       can use this option more than once to specify
			       multiple exclude paths

       --stdexcludes	       exclude temporary files and logs

       --compress	       compress dump file (gzip)

       --storage STORAGE_ID    store resulting files to STORAGE_ID (PVE only)

       --script 	       execute hook script

       --dumpdir DIR	       store resulting files in DIR

       --maxfiles N	       maximal number of backup files per VM.

       --tmpdir DIR	       store temporary files in DIR. --suspend and --stop
			       are using this directory to store a copy of the VM.

       --mailto EMAIL	       send notification mail to EMAIL. You can use
			       this option more than once to specify multiple
			       receivers

       --stop		       stop/start VM if running

       --suspend	       suspend/resume VM when running

       --snapshot	       use LVM snapshot when running

       --size MB	       LVM snapshot size (default 1024)

       --bwlimit KBPS	       limit I/O bandwidth; KBytes per second

       --lockwait MINUTES      maximal time to wait for the global
			       lock. vzdump uses a global lock file to make
			       sure that only one instance is running
			       (running several instance puts too much load
			       on a server). Default is 180 (3 hours).

       --stopwait MINUTES      maximal time to wait until a VM is stopped.

DESCRIPTION

       vzdump is an utility to make consistent snapshots of running virtual machines (VMs). It basically creates a tar archive of the VM private
       area, which also includes the VM configuration files. vzdump currently supports OpenVZ and QemuServer VMs.

       There are several ways to provide consistency:

       "stop" mode
	 Stop the VM during backup. This results in a very long downtime.

       "suspend" mode
	 For OpenVZ, this mode uses rsync to copy the VM to a temporary location (see option --tmpdir). Then the VM is suspended and a second
	 rsync copies changed files. After that, the VM is started (resume) again. This results in a minimal downtime, but needs additional space
	 to hold the VM copy.

	 For QemuServer, this mode work like "stop" mode, but uses suspend/resume instead of stop/start.

       "snapshot" mode
	 This mode uses LVM2 snapshots. There is no downtime, but snapshot mode needs LVM2 and some free space on the corresponding volume group
	 to create the LVM snapshot.

BACKUP FILE NAMES

       Newer version of vzdump encodes the virtual machine type and the backup time into the filename, for example

	vzdump-openvz-105-2009_10_09-11_04_43.tar

       That way it is possible to store several backup into the same directory. The parameter "maxfiles" can be used to specify the maximal number
       of backups to keep.

RESTORE

       The resulting tar files can be restored with the following programs.

       vzrestore: OpenVZ restore utility
       qmrestore: QemuServer restore utility

       For details see the corresponding manual pages.

CONFIGURATION

       Global configuration is stored in /etc/vzdump.conf.

	tmpdir: DIR
	dumpdir: DIR
	storage: STORAGE_ID
	mode: snapshot|suspend|stop
	bwlimit: KBPS
	lockwait: MINUTES
	stopwait: MINUTES
	size: MB
	maxfiles: N
	script: FILENAME

HOOK SCRIPT

       You can specify a hook script with option "--script". This script is called at various phases of the backup process, with parameters
       accordingly set. You can find an example in the documentation directory ("hook-script.pl").

EXCLUSIONS (OpenVZ only)
       vzdump skips the following files wit option --stdexcludes

	/var/log/.+
	/tmp/.+
	/var/tmp/.+
	/var/run/.+pid

       You can manually specify exclude paths, for example:

       > vzdump --exclude-path "/tmp/.+" --exclude-path "/var/tmp/.+" 777

       (only excludes tmp directories)

       Configuration files are also stored inside the backup archive (/etc/vzdump), and will be correctly restored.

LIMITATIONS

       VZDump does not save ACLs.

EXAMPLES

       Simply dump VM 777 - no snapshot, just archive the VM private area and configuration files to the default dump directory (usually
       /vz/dump/).

       > vzdump 777

       Use rsync and suspend/resume to create an snapshot (minimal downtime).

       > vzdump --suspend 777

       Backup all VMs and send notification mails to root.

       > vzdump --suspend --all --mailto root

       Use LVM2 to create snapshots (no downtime).

       > vzdump --dumpdir /mnt/backup --snapshot 777

       Backup all VMs excluding VM 101 and 102

       > vzdump --suspend --exclude 101 --exclude 102

       Restore an OpenVZ machine to VM 600

       > vzrestore /mnt/backup/vzdump-openvz-777.tar 600

       Restore an Qemu/KVM machine to VM 601

       > qmrestore /mnt/backup/vzdump-qemu-888.tar 601

SEE ALSO

	   vzrestore(1) qmrestore(1)

AUTHOR

       Dietmar Maurer <dietmar@proxmox.com>

       Many thanks to Proxmox Server Solutions (www.proxmox.com) for sponsoring this work.

COPYRIGHT AND DISCLAIMER

       Copyright (C) 2007-2009 Proxmox Server Solutions GmbH

       Copyright: vzdump is under GNU GPL, the GNU General Public License.

       This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
       the Free Software Foundation; version 2 dated June, 1991.

       This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

       You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation,
       Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA.

1.2								    2012-04-07								 vzdump(1)
All times are GMT -4. The time now is 04:30 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy