Parallel processing for functions in xargs


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parallel processing for functions in xargs
# 1  
Old 05-17-2015
Parallel processing for functions in xargs

I have a script (ksh) which tries to run a function in parallel for performance gains. I am also trying to limit the number of parallel child processes to avoid overloading the system by using a variable to count triggered processes and waiting for completion e.g.

Code:
do_something () 
{
...
}

process_cnt=0
while 
do
  if [ $process_cnt -eq 3 ]; then 
    wait
  fi
  do_something &
  let process_cnt=process_cnt+1
done < param_file
wait

The problem with this approach is that it is only as good as the worst performer in a batch, i.e. the wait command will wait for all child processed in a batch to finish before triggering the next batch. There is a more flexible way of doing this by using xargs --max-procs command, but unfortunately that seems to be built for shell commands only.

Is there a way I can use xargs to trigger user-defined functions in parallel?

Last edited by vbe; 05-17-2015 at 02:17 PM.. Reason: code tags not icode
# 2  
Old 05-17-2015
Please use CODE tags (it has an icon saying: CODE), not ICODE for multiline code.
You're around here for ~3 years, you should know the difference.

It doesnt matter if its a file, a function or a string.

How to run and 'manage' paralell commands has been discusses quite a few times, please use the search function. (run background jobs/task/scripts/commands paralell)
There are also great examples available.

Specificly in this closed thread: [BASH] Script to manage background scripts (running, finished, exit code)
But there are up to 5 pages of search results using combination of the above search words.

hth

Last edited by sea; 05-17-2015 at 02:57 PM..
This User Gave Thanks to sea For This Post:
# 3  
Old 05-17-2015
Use GNU Parallel

Install GNU Parallel and do this:
Code:
$ foo() { echo $*; }                                          

$ export fun="`typeset -f`"; parallel 'eval "$fun";'foo ::: works
works

$ export fun="`typeset -f`"; parallel 'eval "$fun";'foo :::: param_file


Last edited by tange; 05-17-2015 at 04:59 PM.. Reason: works in zsh, too, with "
# 4  
Old 05-17-2015
Quote:
Originally Posted by sea
Please use CODE tags (it has an icon saying: CODE), not ICODE for multiline code.
You're around here for ~3 years, you should know the difference.

It doesnt matter if its a file, a function or a string.

How to run and 'manage' paralell commands has been discusses quite a few times, please use the search function. (run background jobs/task/scripts/commands paralell)
There are also great examples available.

Specificly in this closed thread: [BASH] Script to manage background scripts (running, finished, exit code)
But there are up to 5 pages of search results using combination of the above search words.

hth
I have been out for a while, and tags seem to have changed since then. The way I remember it, this site used to be about people responding to questions instead of pointing out duplicity. I will have a look at the links that you posted, and will see if that works, but honestly speaking - a suggestion to "search for the right answer" is not really something that strikes me as useful (its not something I would ever do).
# 5  
Old 05-17-2015
Why would one to repeat oneself if its already been said?
Sometimes people (me included) are too focused on the words (or specific tools, your case) they have in their mind, rather than the situation.

There are not too many different ways ways on how to run X numbers of background jobs, and this year this is at least the 3rd thread of its kind, having the same requierments.

One could ask: did you google?
One could ask: did you search the forum?
IMO that is a very legit question with the aim to solve your own question.

I like to help others, even provide tools (with the aim) to help others.
But i dont like to repeat myself, so i redirect to whats already been said, if i'm aware of it (which applies in this case).

Sorry if that hurts your feelings.
Have a nice evening.

EDIT:
A forum is NOT "set and forget".
These 3 Users Gave Thanks to sea For This Post:
# 6  
Old 05-17-2015
Quote:
Originally Posted by jawsnnn
I have been out for a while, and tags seem to have changed since then. The way I remember it, this site used to be about people responding to questions instead of pointing out duplicity. I will have a look at the links that you posted, and will see if that works, but honestly speaking - a suggestion to "search for the right answer" is not really something that strikes me as useful (its not something I would ever do).
Moderator's Comments:
Mod Comment Since you have been out for a while, maybe you should review the rules you agreed to when you joined the UNIX & Linux Forums. They can be found here. Especially note rule #5: Search the forums database with your keywords before asking questions.

You should not expect the volunteers who respond to questions raised in this forum to answer the same questions dozens (or even thousands) of times because the thousands of members in this forum with questions think they all deserve a personal response and are unwilling to search to see if their question has been asked and answered before.

Instead of chastising sea for not wanting to repeatedly discuss a topic that has been addressed many times before, maybe you should thank him for giving you a pointer to a thread that contains a lengthy discussion that seems to be VERY closely related to your question.
These 3 Users Gave Thanks to Don Cragun For This Post:
# 7  
Old 05-17-2015
Quote:
Originally Posted by Don Cragun
Moderator's Comments:
Mod Comment Since you have been out for a while, maybe you should review the rules you agreed to when you joined the UNIX & Linux Forums. They can be found here. Especially note rule #5: Search the forums database with your keywords before asking questions.

You should not expect the volunteers who respond to questions raised in this forum to answer the same questions dozens (or even thousands) of times because the thousands of members in this forum with questions think they all deserve a personal response and are unwilling to search to see if their question has been asked and answered before.

Instead of chastising sea for not wanting to repeatedly discuss a topic that has been addressed many times before, maybe you should thank him for giving you a pointer to a thread that contains a lengthy discussion that seems to be VERY closely related to your question.
I appreciate the pointer to the updated code tags by sea. And I also agree with his later comment that mentions that I may have been fixated on a specific tool for my solution. However...

I will not appreciate a pointer to "google it" or "search the forums" with a thanks. Thats as good as "look at the man pages". Technically true, but practically? No. That implies I did not do my research, and makes assumptions about ME. I have a very specific question, and that question has a specific answer (that I am going to detail below). Whether or not something that has already been posted in a forum that is sprawling and growing by the second is irrelevant and wastes everyone's time. I completely disagree with rule # 5.

Why do I have that inclination? Because I have been on the other end a few times. I see a question that is posted and I try to answer that based on my knowledge. That not only helps the specific person who asks, but also reiterates my experience with that stuff. It may not be the most efficient way of resolving questions, but it is the quickest - which is why I used to visit these forums so often (and which is why countless others do). I never felt burdened by a repeat question... because I knew that the person asking the question was a different person, who may be tried and did not get what he needed from google or other resources.

That - in a nutshell - is why I reacted aversely to the original post. Back to OT:

Code:
#!/bin/ksh
typeset -f fun
function func
{
 echo $1
}

seq 10 | xargs -I% --max-procs=5 ksh -c "$(typeset -f func)"';func "%"'

The code above allows you to pass a function as a parameter to xargs. The key is declaring the function within the ksh -c string using "typeset -f".

I came to realize that implementing this within my solution would require too much complexity for a small performance gain. I would use parallel but that requires installation of the utility (not an option for me unfortunately.). I am going with my original approach for now.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parallel processing

I have 10,000 + files, each of which I need to zip using bzip2. Is ti possible to use bash to create 8 parallel streams sending a new file to be processed from the list when one of the others has finished? (1 Reply)
Discussion started by: garethsays
1 Replies

2. Shell Programming and Scripting

Pass parameters to a function and running functions in parallel

Hi , I have a script which is using a text file as I/P. I want a code where it reads n lines from this file and pass the parameters to a function and now this script should run in such a way where a function can be called in parallel with different parameters. Please find below my script, it... (1 Reply)
Discussion started by: Ravindra Swan
1 Replies

3. Shell Programming and Scripting

Calling multiple functions in parallel

Hello, I have multiple functions within a shell script. eg. function_database_backup, unix_tar_creation, etc. I would like to run these functions in parallel, as each is independent of the other. If these were simple commands, I could have probably run each of the commands in background. ... (1 Reply)
Discussion started by: neil.k
1 Replies

4. Programming

Algorithms for Parallel Processing

Hey, I just wanted to know how many algorithms there are that cannot be accelerated by parallel processing. I know one such algorithm is Euclid's Algorithm (for GCF). does anyone know any other algorithms that cannot be accelerated by pp? if so please list the names and a general sentence of what... (2 Replies)
Discussion started by: azar.zorn
2 Replies

5. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

6. Shell Programming and Scripting

Using xargs for multiple functions

Hi Experts, I am trying to parse some syslog outputs into a separate file per node using the below syntax but am having issues when it comes to my Xargs statements. The command which I was intending on using was: cat syslogs | nawk '/From/ { print $3 }' | uniq | xargs -I {} grep {}... (5 Replies)
Discussion started by: krypton
5 Replies

7. Shell Programming and Scripting

parallel processing

hi i am preparing a set of batches for a set of files sequentially There is a folder /xyz where all the files reside now all the files starting with 01 - will be appended for one below other to form a batch batch01 then all the files starting with 02 - will be appended for one below other to... (7 Replies)
Discussion started by: mad_man12
7 Replies

8. Shell Programming and Scripting

Need Help With Parallel Processing

Hi I am looking for some kind of feature in unix that will help me write a script that can invoke multiple processes in parallel. And make sure that the multiple parallel processes complete successfully before I proceed to the next step. Someone suggested something called timespid or... (6 Replies)
Discussion started by: imnewtothis23
6 Replies

9. Shell Programming and Scripting

parallel processing

Hi I want to run two shell script files parallely. These two scripts are interacting with the database. can any body help on this Pls Regards Audippa naidu.M (3 Replies)
Discussion started by: audippa
3 Replies

10. UNIX for Dummies Questions & Answers

How to do parallel processing??

Hi All, I am working on solaris 8 sparc machine with 2 cpu. I am trying to run my application which generates files. I run multiple instance of the application, but the results don't seem to show as if it were runing parallely. When i run the application once it takes 12 secs to generate a... (1 Reply)
Discussion started by: zing
1 Replies
Login or Register to Ask a Question