script for multi-threaded bash processes


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting script for multi-threaded bash processes
# 1  
Old 08-24-2011
script for multi-threaded bash processes

hey everyone,

I'm having some trouble breaking down some code. It's simple a control script that takes machines meant to be backed up from a list. Then according to that will run multi-threaded processes up until the specified thread limit.

for example if there are 4 machines to be backed up, and you specify the --threads=3 if will do the first 3, wait for one to finish before running the 4th.

Here's my problem. I know what this code does, but not how it works. Any help in breaking this down would be greatly appreciated!^_^

Code:
concurrent=`echo $1 | sed -e 's/--threads=//'`
maxprocs=`cat $serverset | wc -l`
tick=2

for ((i=0; i<$concurrent; i+=1 )); do running[$i]=123456789; done
ran=0
until
        while [ $ran -lt $maxprocs ]; do
                for ((p=0; p<$concurrent; p+=1 )); do
                        proc=${running[$p]}
                        ps -p $proc | fgrep $proc >/dev/null
                        if [ $? -ne '0' ] ; then
                                runproc $ran &
                                running[$p]=$!
                                ((ran+=1))
                                if [ $ran -ge $maxprocs ]; then break 1; fi
                        fi
                done
                sleep $tick
        done
        sleep $tick
do [ `jobs -r|wc -l` -eq 0 ]
done
wait

# 2  
Old 08-26-2011
Code:
# get maximum number of thread slots
concurrent=`echo $1 | sed -e 's/--threads=//'`
# get number of tasks to be done
maxprocs=`cat $serverset | wc -l`
# wait time between two task checks
tick=2

# initialize running info to a value which cannot be a valid PID
for ((i=0; i<$concurrent; i+=1 )); do running[$i]=123456789; done
# Zero tasks are started at the beginning
ran=0
# LOOP until no more running background jobs are present (see condition below)
until
    # while not all tasks are started
    while [ $ran -lt $maxprocs ]; do
        # loop over all thread slots
        for ((p=0; p<$concurrent; p+=1 )); do
            # get running PID info of the thread slot
            proc=${running[$p]}
            # check if a process with that PID is (still) running
            # in the init case this will result in "not running"
            ps -p $proc | fgrep $proc >/dev/null
            # if the thread slot is in state "not running"
            if [ $? -ne '0' ] ; then
                # occupy the thread slot with a background job ...
                runproc $ran &
                # ... and remember its PID
                running[$p]=$!
                # increase started tasks count
                ((ran+=1))
                # if all tasks to be done are started, leave thread slot loop
                if [ $ran -ge $maxprocs ]; then break 1; fi
            fi
        done
        sleep $tick
    done
    sleep $tick
# LOOP until no more running background jobs are present
do [ `jobs -r|wc -l` -eq 0 ]
done
# to be sure: final wait for complete finish of all child processes
wait

# 3  
Old 08-27-2011
awesome! thanks for the help!^_^
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

(bash) Script Processes in Parallel

Hello all, I tried to parralise my treatments but after a while 'ps -ef' display all child process <defunct> (zombie) Parent bash script to process all files (>100000) in directory: for filename in /Data/*.txt; do ./child_pprocess.sh $filename & done exit(0)I understand that the... (1 Reply)
Discussion started by: namnetes
1 Replies

2. Programming

Deallocating memory in multi-threaded environment.

I'm having a hard time figuring out how to manage deallocation of memory in multithreaded environments. Specifically what I'm having a hard time with is using a lock to protect a structure, but when it's time to free the structure, you have to unlock the lock to destroy the lock itself. Which will... (5 Replies)
Discussion started by: gngrwzrd
5 Replies

3. UNIX for Dummies Questions & Answers

Bash: Regulating the number of processes a script can spawn

I've been working on some scripts in which I spawn some background processes. I'd like to be able to limit the number of processes, but have my script spawn additional processes as previous tasks finish. So, let's say I have 20 tasks to complete. Any given task could take from 1 to 10 minutes. ... (7 Replies)
Discussion started by: treesloth
7 Replies

4. Programming

multi-threaded memory leak

Hello All : I write a .c program to test the exactually resource the memory leak as follows: 1 #include <stdio.h> 2 #define NUM 100000 3 void *Thread_Run(void * arg){ 4 //TODO 5 //pthread_datch(pthread_self()); 6 int socket= (int)arg; 7 ... (1 Reply)
Discussion started by: aobai
1 Replies

5. Linux

Multi-threaded encryption @ Fedora 11

Hello, are any of the encryption programs capable of true multi-threading ? Friend of mine tells me that he's been running some testing on Fedora 11 and that the kernel doesn't support multi-threading at that level. I've been looking into TrueCrypt, encfs and both calm to support... (1 Reply)
Discussion started by: TehOne
1 Replies

6. UNIX for Advanced & Expert Users

Multi-threaded encryption @ Fedora 11

Hello, are any of the encryption programs capable of true multi-threading ? Friend of mine tells me that he's been running some testing on Fedora 11 and that the kernel doesn't support multi-threading at that level. I've been looking into TrueCrypt, encfs and both calm to support... (0 Replies)
Discussion started by: TehOne
0 Replies

7. Shell Programming and Scripting

In need of multi threaded perl assistance

I need to write a perl script to execute external programs and grab the output and return code. Each program should be killed if it has not completed within X seconds. Imagine that the script goes something like this : @commands = &get_commands(); foreach $cmd (@commands) { $pid =... (4 Replies)
Discussion started by: SandmanCL
4 Replies

8. AIX

multi threaded program is hanging

I have a Multithreaded program which is hanging on AIX. OS Version: AIX 5.2 and thread library version : 5.2.0.75 We Initiate the process with 50 threads..when we are disconnecting from the process it hangs.There is lots of other stuff involved here.I am just sending the piece of the problem with... (0 Replies)
Discussion started by: hikrishn
0 Replies

9. Programming

HOWTO: Calculate the balance of work in multi-threaded app.

I was wondering if anyone could give me a good idea how to calculate how balanced the threading is on a multi-threaded application. I want a percentage, such as "threads are 80% balanced." This is the way I am currently going about it, maybe it is good, maybe not. First, whenever a thread... (2 Replies)
Discussion started by: DreamWarrior
2 Replies

10. Programming

multi-threaded server, pthreads, sleep

I am trying to writa a multi-client & multi-threaded TCP server. There is a thread pool. Each thread in the pool will handle requests of multiple clients. But here I have a problem. I find a solution but it is not how it must be... i think. When threads working without sleep(1) I can't... (0 Replies)
Discussion started by: Parahat Melayev
0 Replies
Login or Register to Ask a Question