Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Xargs and rsync not sending all files
Top Forums Shell Programming and Scripting Xargs and rsync not sending all files Post 303039517 by Chubler_XL on Tuesday 8th of October 2019 04:15:36 PM
Old 10-08-2019
It could be some sort of resource issue, or connection limit. I'd try less threads.

With disk and network I/O bottlenecks involved, I find it hard to believe that 40 threads would be quicker than 5, on my low spec system here 5 takes about twice as long as 1!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

MV files with xargs or -exec

Hi I need to move multiple (say 10 files) from one location to another location. My selection would be like this... ls -ltr *.arc | head ---> Need to move top 10 files with single command without iterating in loop. I know we can move files like this with find command but not sure if I can... (4 Replies)
Discussion started by: malaymaru
4 Replies

2. Shell Programming and Scripting

find with xargs to rm found files

I believe what is happening is rm is executing in the script on every directory and on failure of the first it stops although returns status 0. find $HOME -name /directory/filename | xargs -l rm This is the code I use but file remains. I am using sun solaris system which has way limited... (4 Replies)
Discussion started by: Ebodee
4 Replies

3. Shell Programming and Scripting

move files using xargs

Hi, I have many files like below in my currect dir test1.me test2.me test3.me I wanted them to rename without .me extention I tried below find . -name "*.me" | xargs -i mv {} `echo {} | sed 's/\.me//'` It thorws error that cant mv ./test1.me to ./test1.me as they are... (6 Replies)
Discussion started by: sunilmenhdiratt
6 Replies

4. Shell Programming and Scripting

help using find/xargs to apply mp3gain to files

I need to apply mp3gain (album mode) to all mp3 files in a given directory. Each album is in its own directory under /media/data/music/albums for example: /media/data/music/albums/foo /media/data/music/albums/bar /media/data/music/albums/more What needs to happen is: cd... (4 Replies)
Discussion started by: audiophile
4 Replies

5. Shell Programming and Scripting

Fast processing(mv command) of 1 million+ files using find, mv and xargs

Hi, I'd like to ask if anybody can help improve my code to move 1 million+ files from a directory to another: find /source/dir -name file* -type f | xargs -I '{}' mv {} /destination/dir I learned this line of code from this forum as well and it works fine. However, file movement is kinda... (6 Replies)
Discussion started by: agentgrecko
6 Replies

6. Shell Programming and Scripting

How to copy files from one location to another using xargs??

Hello Experts, I need to copy files from one location to another using xargs. Tried something like this (In Ubuntu & Solaris ). mkdir -p 1234; find /home/emd/Desktop/n007/M007/ -type f -name "A2014*" | xargs -0 cp -r {} /home/emd/Desktop/1234 But every time i run this, a weird error... (6 Replies)
Discussion started by: Saidul
6 Replies

7. Shell Programming and Scripting

How to delete directories and files with xargs?

Hello, i have an dynamical apache_cache that I need to clean all days (older tant one day) with an unix command : find /usr/IBM/HTTPServer/apache_cache/ -name '*' -mtime +1 -print0|xargs -0 rm -r -- but it didn't work. Could you explain me why. So I will put below all my script :... (13 Replies)
Discussion started by: steiner
13 Replies

8. Shell Programming and Scripting

Using rsync with xargs

I want to run parallelise rsync with xargs. The plan is to take separate directories and run rsync in parallel. find . -mindepth 1 -maxdepth 1 print0 | xargs -n1 -P0 rsync -avz Hoxever I am stuck on how to use rsync in this way ---------- Post updated at 04:44 AM ---------- Previous... (3 Replies)
Discussion started by: kristinu
3 Replies

9. Shell Programming and Scripting

Linux command rsync sending files incrementally

Please, i have a question about rsync command: Here is the command that i have used in my script: rsync -ratlz --rsh="/usr/bin/sshpass ssh -o StrictHostKeyChecking=no" -aAXHltzh --progress --numeric-ids --devices --rsync-path="rsync" $1 $hostname:$1 using this command, i can... (0 Replies)
Discussion started by: chercheur111
0 Replies

10. Shell Programming and Scripting

Xargs to call python executable to process multiple bam files

I am running the below loop that to process the 3 bam files (which isn't always the case). A .py executable is then called using | xargs sh to further process. If I just run it with echo the output is fine and expected, however when | xargs sh is added I get the error. I tried adding | xargs... (4 Replies)
Discussion started by: cmccabe
4 Replies

LEARN ABOUT HPUX

nfs4_max_threads

nfs4_max_threads(5)						File Formats Manual					       nfs4_max_threads(5)

NAME

       nfs4_max_threads - control the number of kernel threads that perform asynchronous I/O for the NFS version 4 client

VALUES

   Failsafe
   Default
   Allowed values
   Recommended values
       A  warning will be issued at runtime if the tunable is set to a value greater than 256 threads since this is beyond the tested limit.  This
       is not a serious warning but just an information message for the administrator.

DESCRIPTION

       controls the number of kernel threads that perform asynchronous I/O for the NFS version 4 client. Since NFS is based  on  RPC  and  RPC	is
       inherently  synchronous,  separate execution contexts are required to perform NFS operations that are asynchronous from the calling thread.
       The operations which can be executed asynchronously are read for read-ahead, readdir for readdir read-ahead,  and  write  for  putpage  and
       pageio requests.

   Who Is Expected to Change This Tunable?
       The distributed file system administrator should examine this value depending on the network bandwidth available.

   Restrictions on Changing
       The  tunable  is  dynamic;  however, the number of threads for a file system is set when the file system is mounted. To affect a particular
       file system, unmount and mount the file system after changing this parameter.

   When Should the Value of This Tunable Be Raised?
       If the network is very high bandwidth and the client and server have sufficient resources, increase this value to more effectively  utilize
       the available network bandwidth, the client resources, and the server resources.

   What Are the Side Effects of Raising the Value?
       An  increase in the system resources and the possibility that (see nkthread(5)) may need to be increased. The total number of  asynchronous
       threads for NFS version 4 mount points cannot exceed 20% of the available threads defined by NFS mounts will fail if the NFS mount  command
       cannot guarantee the ability to create the maximum number of threads for that mount point.

   When Should the Value of This Tunable Be Lowered?
       In a very low bandwidth network, decrease this value so that the NFS client does not overload the network.

   What Are the Side Effects of Lowering the Value?
       Decreasing this value may impact NFS performance.

WARNINGS

       All  HP-UX kernel tunable parameters are release specific.  This parameter may be removed or have its meaning changed in future releases of
       HP-UX.

       Installation of optional kernel software, from HP or other vendors, may cause changes to tunable  parameter  values.   After  installation,
       some  tunable  parameters may no longer be at the default or recommended values.  For information about the effects of installation on tun-
       able values, consult the documentation for the kernel software being installed.	For information about optional kernel  software  that  was
       factory installed on your system, see at

AUTHOR

       was developed by Sun Microsystems, Inc.

SEE ALSO

       kctune(1M), sam(1M), gettune(2), settune(2), nfs2_max_threads(5), nfs3_max_threads(5), nkthread(5).

							     Tunable Kernel Parameters					       nfs4_max_threads(5)
All times are GMT -4. The time now is 06:49 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy