[BASH] Script to manage background scripts (running, finished, exit code)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting [BASH] Script to manage background scripts (running, finished, exit code)
# 1  
Old 10-16-2014
[BASH] Script to manage background scripts (running, finished, exit code)

Heyas,

Since this question (similar) occur every now and then, and given the fact i was thinking about it just recently (1-2 weeks) anyway, i started to write something Smilie

The last point for motivation was How to run scripts parallely inside shell script?
Where I've posted a screen-shot from my very first (and small) approach.

Now i want to write a script that will work for/with:
  • 'any' amount of scripts passed,
  • provides a limiter, so only X scripts are running simultaneously
  • Prints the status code of each of the scripts ran

So I had it running and working, but then figured, I've only tested 4 scripts (sleep [5-30secs), but the LIMIT was 5, so I reduced the limit to 2, now i cant figure why its not working.
Actually, it wont even run (properly) with the default LIMIT of 5 even when only 4 scripts are passed Smilie
And the previous code no longer works neither, since I've renamed some of the variables, for a better 'sort'.

My approach is:
  • scripts_remains = array of arguments (the scripts passed)
  • scripts_start = array is filled only temporary, as long the counter scripts_running is less than the LIMIT.
  • While filling the scripts_start, i remove the according array element from scripts_remains.
  • scripts_todo = array that gets filled as soon the scripts_start entry is started. (and afterwards removed from scripts_start)
  • scripts_id = array which holds the PID of the script of the same ID of scripts_todo
When the script no longer finds the PID, it moves the element from scripts_todo/scripts_id to done_scripts, and fills done_ret, both with the same array id.
(this part works still/again)

But the starting array gets only the first script and not more... however, it does list as many entries as the LIMIT allows, but all contain the first script and only 1 entry does actually show the proper PID.
scripts_remains still holds all array elements, which confuses me, furthermore scripts_todo holds only 1 array element. but lists as many entries as the LIMIT allows.

NOTE: This script is(will be) part of my TUI (text user interface) package, therefore all those tui-* commands ARE available and 'valid' - on my system at least.

Please see the screen-shot on how it outputs, and the code segment that is faulty.
Acknowledge, that the error referring to line 332 is due to the wrong set / missing array values.

I have a small script, that is working, but since i want/need to have some handlers and argument handling around it,
it does in no way refer the issues i experience.

I'm asking if you either see a 'wrong'-code, in the part visible, or if you would have an advice for another approach/method to achieve my goal?
Thank you in advance

EDIT:
It seems that the first script (only one loaded to array) is started/executed as many times as the limit is set to, without continuing to the next array element.. Smilie
But i cant pinpoint it.

If you need the full code, i'd happily share it, just since it wont run (unless you have TUI installed) i thought its not really helpfull anyway.

EDIT:
Done the 'no limit' part, which works:
Code:
			while [[ ! -z "${scripts_remains[@]}" ]]
				do	# Start the script:
					script="${scripts_remains[$counter_start]}"
					scripts_todo[$counter_start]="$script"
					
					# Generate the command
					[[ [./] = "${script:0:1}" ]] && PRE="" || PRE="./"
					cmd="${PRE}\"$script\" ; echo \$? > \"$TEMP/$script.ret\""
					#doLog "Executing: $cmd"
					
					# Get PID to array
					( eval $cmd ) &
					scripts_id[$counter_start]="$!"
					
					# Remove it from array, and move to next entry
					unset scripts_remains[$counter_start]
					((counter_start++))
				done

So I need to get this code segment working with a LIMIT factor, so only LIMIT amount of scripts are started as long scripts_running is smaller than the LIMIT - see the main post -

Screen-shot *1*small : the non-working part
Screen-shot *2*small : the working no-limit part

EDIT3:
I'm now 100% sure, I've had somewhere screwed up with the arrays, but atm I'm too blind...
[BASH] Script to manage background scripts (running, finished, exit code)-tui-bg-scripts-1_smalljpg
[BASH] Script to manage background scripts (running, finished, exit code)-tui-bg-scripts-2_smalljpg

Last edited by rbatte1; 10-17-2014 at 07:35 AM.. Reason: Added LIST tags
# 2  
Old 10-16-2014
This is bad:

Code:
while [[ ! -z "${scripts_remains[@]}" ]]

If there's more than one thing in that array, this will die with 'too many arguments' or 'unexpected argument' or the like for cramming [[ ! -z "a" "b" "c" "d" "..." ]] into your statement.

Try while [[ ! -z "${scripts_remains[*]}" ]]

I might be able to comment more on your code if you posted anything except screenshots.

If it's too big to post, trim it down! Make a minimal example that still shows the problem. There's probably good odds that doing so will actually find the problem, too.

I'd be tempted to just use strings and/or positional parameters instead of arrays here, too. They're much easier to deal with in a lot of circumstances.

Code:
# loop over array
for X in "${ARRAY[@]}" ; do ...  ; done

# loop over string
for X in $string ; do ... ; done

# rotating an array
TMP=${ARR[0]}
for((N=1; N<${#ARR[@]}; N++)) ; do ARR[$((N-1))]=ARR[$N] ; done ; ARR[${#ARR[@]}]=$TMP

# rotating a string
set -- $string ; set -- "$@" $1 ; shift
string="$*"


Last edited by Corona688; 10-16-2014 at 01:24 PM..
# 3  
Old 10-16-2014
Refinition of the problem:
It is now starting script1 - three times at once, while removing (but only once) script2, but then never again any other script from the remains list.
As well as adding another tripple bunch script1's as soon the previous three script1 ones finsihed, appending them infitie to done list..

Right, minimal example... working on...

EDIT:
In the screenshot *4_small* you see again, the '4' within an error messsage - but labled as /script1 - both a path&name issue, but in the screenshot *3_small* the 4th script is not listed at all.
[BASH] Script to manage background scripts (running, finished, exit code)-tui-bg-scripts-3_smalljpg
[BASH] Script to manage background scripts (running, finished, exit code)-tui-bg-scripts-4_smalljpg

Last edited by sea; 10-16-2014 at 03:19 PM.. Reason: Code fix & image screenshot as illustation
# 4  
Old 10-16-2014
Ok, here's the smallest i could get...
But now script3 gets lost among the way, everything else seems to be working fine...

Code:
#!/bin/bash
#
#	Variables
#
	scripts_remains=( "${@}" )
	scripts_total=${#scripts_remains[@]}
	TMP_DIR=$HOME/.cache/$$
	
	# Filled in the process
	unset scripts_todo[@] scripts_id[@]
	unset done_scripts[@] done_ret[@]
	
	# Defaults
	LIMIT=5
	WAIT=5
	
	# Counters - Fixed
	counter_done=0
	
	# Counters - Dynamic
	counter_start=0
	counter_running=0
#
#	Environment check
#
	[[ -d "$TMP_DIR" ]] || mkdir -p "$TMP_DIR"
#
#	Display & Action --> limit 5, passing 4
#
	while [[ $counter_done -lt $scripts_total ]]
	do	# Loop the menu & Reset some values
		
		# Step 1
		# Check if there are files to be started
		echo "Scripts @ start"
# The Limit check is worthless, even when set to 2, all passed scripts gets executed on first loop...
		if [[ $counter_running -lt $LIMIT ]]
		then	# So we look in the script_remains for tasks
			num=0
			for S in ${scripts_remains[@]};do
				[[ -z "$S" ]] && break
				# Generate the command & save to new array
				[[ [./] = "${S:0:1}" ]] && PRE="" || PRE="./" 
				cmd="$PRE$S ; echo \$? > $TMP_DIR/$S.tmp"
				echo "Starting: $S"
				(eval $cmd) &
				scripts_id[$counter_start]="$!"
				scripts_todo[$counter_start]="$S"
				unset scripts_remains[$num]
				((counter_start++))
				((counter_running++))
				((num++))
			done
		fi
		
		# Step 2
		# Print status of already done scripts
		echo "Scrips @ done"
		C=0
		for D in "${done_scripts[@]}";do
			R=${done_id[$C]}
			echo "$D ended $R"
			((C++))
		done
		
		# Step 3
		# Show current tasks running -- now loops here endlessly...  because a script gets lost within the loop
		num=0
		echo "Scripts @ work"
		for W in "${scripts_todo[@]}";do
			# Only display if array element is not empty
			if [[ ! -z "$W" ]]
			then	val=${scripts_id[$num]}
				if [[ ! -z "$val" ]]
				then	if ps -ha | grep $val|grep -v -q grep
					then	echo "$W works : $val"
					else	echo "$W has ended..."
						# Unset this item now
						done_scripts[$counter_done]="$W"
						read R < $TMP_DIR/$W.tmp
						done_id[$counter_done]="$R"
						unset scripts_todo[$num] scripts_id[$num]
						((counter_done++))
						((counter_running--))
					fi
				fi
				((num++))
			fi
			
		done
		
		[[ $counter_done -lt $scripts_total ]] && \
			echo "wait for update: $WAIT" && \
			sleep $WAIT && \
			clear
	done
#
#	Clean up temp files
#
	rm -fr "$TMP_DIR"

BTW: These are my test scripts
Code:
grep -n sleep *
script1:2:sleep 30 
script2:2:sleep 9
script3:2:sleep 20
script4:2:sleep 15


Last edited by sea; 10-16-2014 at 04:29 PM..
# 5  
Old 10-16-2014
Quote:
Originally Posted by sea
Ok, here's the smallest i could get...
But now script3 gets lost among the way
What does 'lost along the way' mean?
# 6  
Old 10-16-2014
Its no longer visible. File still exists physicly on the disk, but it gets lost somewhere within the arrays...

As in:
Code:
+ ~/tmp/9388 $ sh ../../psm-mini.sh *
Scripts @ start
Starting: script1
Starting: script2
Starting: script3
Starting: script4
Scrips @ done
Scripts @ work
script1 works : 19364
script2 works : 19365
script3 works : 19367
script4 works : 19369
wait for update: 5
...
some screen updates later
...
Scripts @ start
Scrips @ done
script2 ended 0
script4 ended 0
script1 ended 0
Scripts @ work
wait for update: 5

As soon the first script ends, script3 gets 'lost'

EDIT:
Also, when i change LIMIT to 2, all 4 scripts are started at once.. Smilie
# 7  
Old 10-16-2014
I don't think "unset" really does what you think it does -- deleting from the middle of an array like that. Whenever I do that I end up with "holes", indexes that still exist but have no value. Associative arrays in shell is pretty chancy anyhow, you often won't have a bash new enough, or have bash at all.

Why keep an array of arguments when you already have one, $@

A construct I often use:

Code:
#!/bin/bash

# This is a ring buffer.  Append at PIDS[$PW], read at PIDS[$PR].
# PW increments when a process is added, PR when a process dies,
# and both wrap at MAX.  Order is not important when removing
# since it shuffles the last process to whatever got deleted.
PIDS=()
PW=0
PR=0
MAX=2

running() {     # running "arrayindex"
        ps "${PIDS[$1]}" >/dev/null
}

add() {         # add "pid"
        PIDS[$PW]="$1"
        PW=$(( (PW+1)%MAX ))
        ((TOTAL++))
}

rem() {         # rem "arrayindex"
        # Take PID from the end, plunk it where the one to be deleted is
        PW=$(( (PW-1) % MAX ))
        [[ "$PW" -lt 0 ]] && PW=$((PW+MAX))

        PIDS[$PR]="${PIDS[$PW]}"
        ((TOTAL--))
}

# Start each given program in turn
for X in "$@"
do
        while [[ "$TOTAL" -ge "$MAX" ]]
        do
        for((N=0, I=PR; (TOTAL>=MAX) && (N<TOTAL); N++, I=(I+1)%MAX ))
        do
                if ! running "$I"
                then
                        rem "$I"
                        break
                fi
        done
                [[ "$TOTAL" -ge "$MAX" ]] && sleep 0.1
        done

        # Refers to array indexes, i.e. /tmp/$$-0 for array index 0.
        $X >/tmp/$$-$PW &
        add "$!"
done

while [[ "$TOTAL" -ge "$MAX" ]]
do
        for((N=0, I=PR; N<$TOTAL; N++, I=(I+1)%MAX ))
        do
                if ! running "$I"
                then
                        rem "$I"
                        break
                fi
        done
done

wait
rm -f /tmp/$$-*


Last edited by Corona688; 10-16-2014 at 05:54 PM..
This User Gave Thanks to Corona688 For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Terminal running bash/rsync script does not close with exit (MacOS High SIerra)

Hello, I am running a bash script to do an rsync back on a computer running MacOS High Sierra. This is the script I am using, #!/bin/bash # main backup location, trailing slash included backup_loc="/Volumes/Archive_Volume/00_macos_backup/" # generic backup function function backup {... (12 Replies)
Discussion started by: LMHmedchem
12 Replies

2. Shell Programming and Scripting

Problems running scripts in the background

Hi Could someone offer some help on this problem I've got with running a background process. As part of a script that does a stop/start/status for a piece of software called SAS, the following extract is from part of the start step. My issue is that when the script is run, the control... (0 Replies)
Discussion started by: GavP
0 Replies

3. Shell Programming and Scripting

Running scripts in background

Hi, below is my master script wihch inturn runs 2 scripts in background #master_script.sh ./subscript1.sh & ./subscript2.sh & executed the master_script.sh from unix command prompt $ ./master_script.sh it is executing the subscripts and they are completing fine, however master_script.sh is... (2 Replies)
Discussion started by: JSKOBS
2 Replies

4. Shell Programming and Scripting

Bash Question: HowTo Exit Script with User Input While Process is Running Mid-Loop?

Hi, I have written a script that allows me to repetitively play a music file $N times, which is specified through user input. However, if I want to exit the script before it has finished looping $N times, if I use CTRL+c, I have to CTRL+c however many times are left in order to complete the loop.... (9 Replies)
Discussion started by: hilltop_yodeler
9 Replies

5. Shell Programming and Scripting

Catch exit code of specific background process

Hi all, i hava a specific backgroud process. I have de PID of this process. At some time, the process finish his job, is there any way to catch the exit code? I use "echo $?" normally for commands. Thanks! (2 Replies)
Discussion started by: Xedrox
2 Replies

6. Shell Programming and Scripting

Capturing the exit status of the script running in background

Hi All, I have a scenario where I am executing some child shell scripts in background (using &)through a master parent script. Is there a way I can capture the exit status of each individual child script after the execution is completed. (2 Replies)
Discussion started by: paragkalra
2 Replies

7. Shell Programming and Scripting

Issues with exit after running jobs in background

I have the following sample script to run a script the jobs with the same priority(in this case field3) in parallel; wait for the jobs to finish and run the next set of jobs in parallel.When all the lines are read exit the script. I have the following script which is doing evrything I want... (1 Reply)
Discussion started by: hyennah
1 Replies

8. UNIX for Dummies Questions & Answers

background job finished notification

In my last job someone gave me the command to put in my .profile that let me know when a job I had running in the background finished. It was a word about 5 char long. I can't remember it! (4 Replies)
Discussion started by: nkeller
4 Replies

9. Shell Programming and Scripting

exit status of Invoking two or more scripts in background

I have a sript which is going to trigger other 3 scripts in background simultaneously for eg: Main Script:(main.sh) ----------- sh a.sh & sh b.sh & sh c.sh & How to catch the exit status and store it in a variable for all those three scripts in main script. Is there any other way of... (4 Replies)
Discussion started by: Omkumar
4 Replies
Login or Register to Ask a Question