Sponsored Content
Top Forums Shell Programming and Scripting Xargs and rsync not sending all files Post 303039517 by Chubler_XL on Tuesday 8th of October 2019 04:15:36 PM
Old 10-08-2019
It could be some sort of resource issue, or connection limit. I'd try less threads.

With disk and network I/O bottlenecks involved, I find it hard to believe that 40 threads would be quicker than 5, on my low spec system here 5 takes about twice as long as 1!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

MV files with xargs or -exec

Hi I need to move multiple (say 10 files) from one location to another location. My selection would be like this... ls -ltr *.arc | head ---> Need to move top 10 files with single command without iterating in loop. I know we can move files like this with find command but not sure if I can... (4 Replies)
Discussion started by: malaymaru
4 Replies

2. Shell Programming and Scripting

find with xargs to rm found files

I believe what is happening is rm is executing in the script on every directory and on failure of the first it stops although returns status 0. find $HOME -name /directory/filename | xargs -l rm This is the code I use but file remains. I am using sun solaris system which has way limited... (4 Replies)
Discussion started by: Ebodee
4 Replies

3. Shell Programming and Scripting

move files using xargs

Hi, I have many files like below in my currect dir test1.me test2.me test3.me I wanted them to rename without .me extention I tried below find . -name "*.me" | xargs -i mv {} `echo {} | sed 's/\.me//'` It thorws error that cant mv ./test1.me to ./test1.me as they are... (6 Replies)
Discussion started by: sunilmenhdiratt
6 Replies

4. Shell Programming and Scripting

help using find/xargs to apply mp3gain to files

I need to apply mp3gain (album mode) to all mp3 files in a given directory. Each album is in its own directory under /media/data/music/albums for example: /media/data/music/albums/foo /media/data/music/albums/bar /media/data/music/albums/more What needs to happen is: cd... (4 Replies)
Discussion started by: audiophile
4 Replies

5. Shell Programming and Scripting

Fast processing(mv command) of 1 million+ files using find, mv and xargs

Hi, I'd like to ask if anybody can help improve my code to move 1 million+ files from a directory to another: find /source/dir -name file* -type f | xargs -I '{}' mv {} /destination/dir I learned this line of code from this forum as well and it works fine. However, file movement is kinda... (6 Replies)
Discussion started by: agentgrecko
6 Replies

6. Shell Programming and Scripting

How to copy files from one location to another using xargs??

Hello Experts, I need to copy files from one location to another using xargs. Tried something like this (In Ubuntu & Solaris ). mkdir -p 1234; find /home/emd/Desktop/n007/M007/ -type f -name "A2014*" | xargs -0 cp -r {} /home/emd/Desktop/1234 But every time i run this, a weird error... (6 Replies)
Discussion started by: Saidul
6 Replies

7. Shell Programming and Scripting

How to delete directories and files with xargs?

Hello, i have an dynamical apache_cache that I need to clean all days (older tant one day) with an unix command : find /usr/IBM/HTTPServer/apache_cache/ -name '*' -mtime +1 -print0|xargs -0 rm -r -- but it didn't work. Could you explain me why. So I will put below all my script :... (13 Replies)
Discussion started by: steiner
13 Replies

8. Shell Programming and Scripting

Using rsync with xargs

I want to run parallelise rsync with xargs. The plan is to take separate directories and run rsync in parallel. find . -mindepth 1 -maxdepth 1 print0 | xargs -n1 -P0 rsync -avz Hoxever I am stuck on how to use rsync in this way ---------- Post updated at 04:44 AM ---------- Previous... (3 Replies)
Discussion started by: kristinu
3 Replies

9. Shell Programming and Scripting

Linux command rsync sending files incrementally

Please, i have a question about rsync command: Here is the command that i have used in my script: rsync -ratlz --rsh="/usr/bin/sshpass ssh -o StrictHostKeyChecking=no" -aAXHltzh --progress --numeric-ids --devices --rsync-path="rsync" $1 $hostname:$1 using this command, i can... (0 Replies)
Discussion started by: chercheur111
0 Replies

10. Shell Programming and Scripting

Xargs to call python executable to process multiple bam files

I am running the below loop that to process the 3 bam files (which isn't always the case). A .py executable is then called using | xargs sh to further process. If I just run it with echo the output is fine and expected, however when | xargs sh is added I get the error. I tried adding | xargs... (4 Replies)
Discussion started by: cmccabe
4 Replies
aio_proc_thread_pct(5)						File Formats Manual					    aio_proc_thread_pct(5)

NAME
aio_proc_thread_pct - percentage of all process threads allowed in AIO pool VALUES
Failsafe Default Allowed values Recommended values DESCRIPTION
The implementation of POSIX AIO on HP-UX uses kernel threads to perform I/Os to filesystems that do not directly support true asynchronous I/O. (This distinction is transparent to the user.) The kernel threads are organized into worker-thread pools (called AIO thread pools) created on a per-process basis. Since a thread pool mechanism for I/Os introduces a variety of trade-offs concerning utilization of CPU time vs. I/O resources, four dynamic tunables are available to customize the behavior of this thread pool: aio_proc_threads(5), aio_proc_thread_pct(5), aio_req_per_thread(5), and aio_monitor_run_sec(5). Please see individual manpages for details on each of these tunables. The tunable specifies, on a per-process basis, the percentage of threads that can be used by the POSIX AIO system as kernel threads for issuing I/Os. The percentage is taken as a percentage of which is the upper bound on the number of threads a process may have. This tunable interacts with in the following way: the maximum number of threads used for AIO will be the smaller of the two values defined by the two tunables; i.e.: This allows the number of AIO threads to vary dynamically with but to always be bound by an absolute limit of Who Is Expected to Change This Tunable? System administrators that run applications requiring heavy usage of POSIX AIO to filesystems. Restrictions on Changing This tunables is dynamic. Changes to to this tunable take effect immediately for new processes started after the change. They also impact existing processes, but the speed with which the changes propagate to running processes is determined by the tunable When Should the Value of This Tunable Be Raised? should be raised for applications that do not use very many threads for their own work, but desire high performance from the POSIX AIO sub- system. What Are the Side Effects of Raising the Value of This Tunable? Some applications that use POSIX AIO but also require a large number of threads may find that they are unable to create new threads, because the POSIX AIO thread pool ends up using too many of a process' allowable threads. In addition, using a larger number of kernel threads might lead to increased CPU utilization. When Should the Value of This Tunable Be Lowered? should be lowered when POSIX AIO performance is acceptable but applications using POSIX AIO are seeing errors when trying to create new threads for other work. What Are the Side Effects of Lowering the Value of This Tunable? By ultimately reducing the number of threads available to handle POSIX AIO requests, overall I/O throughput of the POSIX AIO subsystem could be reduced. What Other Tunables Should Be Changed at the Same Time as This One? interacts with this tunable by setting a strict limit on the number of threads that can be used for POSIX AIO. This allows a hard limit to be imposed, regardless of what value happens to take. defines the desired relationship between the number of POSIX AIO kernel threads and the number of I/Os to be serviced. defines how often (in seconds) the AIO thread mechanism will monitor itself for adherence to the constraints defined by the tunables above. WARNINGS
All HP-UX kernel tunable parameters are release specific. This parameter may be removed or have its meaning changed in future releases of HP-UX. Installation of optional kernel software, from HP or other vendors, may cause changes to tunable parameter values. After installation, some tunable parameters may no longer be at the default or recommended values. For information about the effects of installation on tun- able values, consult the documentation for the kernel software being installed. For information about optional kernel software that was factory installed on your system, see at AUTHOR
was developed by HP. SEE ALSO
kctune(1M), sam(1M), gettune(2), settune(2), aio_proc_threads(5), aio_req_per_thread(5), aio_monitor_run_sec(5). Tunable Kernel Parameters aio_proc_thread_pct(5)
All times are GMT -4. The time now is 03:11 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy