UNIX for Advanced & Expert Users

"man select" will tell you all about the select API.

The following script can demonstrate; please test it yourself:

Code:

{
suck_stdin() {
local my_line
read my_line
echo "suck_stdin: $my_line"
}

while read line
do
 echo "passwd: $line"
 suck_stdin
done < /etc/passwd
} < /etc/group

You see that the suck_stdin() function reads a line, therefore it obscures the while loop. Here the suck_stdin draws every second line from /etc/passwd.
Now please harden it with the suggested descriptor magic:

Code:

{
suck_stdin() {
local my_line
read my_line
echo "suck_stdin: $my_line"
}

while read line <&3
do
 echo "passwd: $line"
 suck_stdin
done 3< /etc/passwd
} < /etc/group

and the suck_stdin() won't obscure the while loop - instead it reads from the outer stdin (here: /etc/group).
Commands like ssh behave like the suck_stdin (ssh can usually be fixed with the -n option), and maybe also your m_* functions contain read or ssh.
Conclusion:
without knowing your m_* commands, I suggest to harden your while loop.

The output of strace means really doesn't mean much out of context. Have you followed any of my advice at all yet? Do you know what those file descriptors are?

The manpages contain much information about it.

Consider the bashdb debugger as well. It has some useful features, but requires serious reading beforehand.

BASH with Debugger and Improved Debug Support and Error Handling

Thanks very much guys

That's the kind of stuff I have been looking for.

I'll certainly play around with the code posted and I wasn't aware of the select command, I assumed it was something low level in the kernal maybe

I still don't know what the file descriptors are, as since running under strace it hasn't failed. But if it does I should be able to discover where the problem is.

I don't know if the -n will help with this version of the harness. But if things get bad I'll try it.

The harness uses ssh multiplexing to add the ssh connections to NIX domain sockets and then echos stdout from the remote server into a local report file in real time.

The new version I'm working on uses a messaging framework to send a report file so shouldn't have any issues like this.

I hope

Thanks for the feed back, really helpful stuff.

---------- Post updated at 05:37 PM ---------- Previous update was at 09:50 AM ----------

That's really interesting MIG

I wasn't aware of that behaviour before.

Thanks

---------- Post updated 03-07-14 at 02:18 PM ---------- Previous update was 02-07-14 at 05:37 PM ----------

Thanks to all of you for your help.

The job eventually failed again and I was able to trace the problem to a hanging write to a named pipe that the client process had died on.

I've put in a rather ugly fix that does the write in the background and then kills $! if it is still running.

It would be better if I could test if a process is listening on the end of the pipe but I haven't found a way of doing so.

Does anyone know if that can be done?

Thanks again for all the helpful input

Some Linux ship with a timeout command.
Otherwise this function can be used:

Code:

timeout () {
case $1 in
[1-9]|[1-9][0-9]|[1-9][0-9][0-9]|[1-9][0-9][0-9][0-9])
 to=$1
 shift
 ;;
*)
 echo "usage: $0 timeoutseconds command..."
 return 1
 ;;
esac
perl -e "alarm $to; exec @ARGV" "$@"
# without perl one can try
# ("$@"; kill 0) & (sleep $to; kill 0) &
}

Now, replace your

Code:

hanging_command

by e.g.

Code:

timeout 5 hanging_command

and hanging_command will get a SIGALRM after 5 seconds, that hopefully kills it.

If you can use a non-named pipe, and keep it open, if the client dies, the writer will receive SIGPIPE.