Command timeout from inside script.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Command timeout from inside script.
# 1  
Old 02-01-2012
Command timeout from inside script.

Hi,

I've written a very robust script to get an external IP address from 'behind' a router. It uses many web pages randomly choosing which one/ones to use at run time. The "fetch the web page containing the IP address" is handled by either wget or curl both of which have their 'max time for the operation' option set to 1 second. Most of the web servers respond within a max of 500ms, so the idea of having the timeout set to 1 second is so that if a server is slow to respond the script just moves on to another server.

I've noticed that neither curl nor wget actually keep to the timeout. On my machine this was not a problem, the timeout always kicked in at the latest of about 2.5 seconds. Over the last 24 hours I've tested the script on a Debian machine, having the script run 10 times every 5 mins. Looking at the results I've discovered that on 14 occasions (of the 7560) the timeout failed catastrophically, ranging from 27.722 to 170.643 seconds - the latter is almost 3 mins from a 1 sec timeout.

It seems that I'll have to write my own timeout routine. There are quite a lot of examples out there but the all seem to rely on either using the program 'timeout' (which is not POSIX), or using a timeout script, Eg. timeout_script -t 1 command - which is not what I need at all.

Here's a little bit of my code:

Code:
# curl command line for curl users.
curlCommand="curl --silent --max-time $timeout"

# wget command line for wget users.
wgetCommand="wget --quiet --timeout=$timeout --tries=1 --output-document=-"

urlDownloaderProg=$curlCommand

grepExpr="[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+"

ipAdd=$($urlDownloaderProg "$url" | grep -Eo "$grepExpr" | uniq)

What I need to do is for the line beginning 'ipAdd=' to return immediately, and then for the script to monitor its child process and kill it if it is not finished after 1 second. But I'm not sure how to do this - in particular how do I get the ipAdd= line to return immediately, I had problems using '&' both inside and outside the final ')', while still storing the results of the line in the variable ipAdd. Maybe I am going about this the wrong way and need a different approach.

Any help or advise would be greatly appreciated. Many thanks.
# 2  
Old 02-01-2012
I don't think you can get it to return immediately inside backticks like that. Save its output to a temporary file instead, then use it.

Code:
urlDownloaderProg "$url" > /tmp/$$ &
PID=$! # PID of background command
sleep 1
# If curl has already quit, killing it won't change its exit status
# from 0 into 1.
kill "$PID" 2> /dev/null
if ! wait # curl returned error
then
        echo "timeout" >&2
        rm -f /tmp/$$
        exit 1
fi

ipAdd=$(grep -Eo "$grepExpr" /tmp/$$ | uniq)
rm -f /tmp/$$

This has a side-effect though; the minimum time will be one second, too. If you have Linux, you can do sub-second sleep times:

Code:
urlDownloaderProg "$url" > /tmp/$$ &
PID=$! # PID of background command

END=$((SECONDS+1))

while [[ -d /proc/$PID && "$SECONDS" -lt "$END" ]]
do
        sleep 0.1
done

# If curl has already quit, killing it won't change its exit status
# from 0 into 1.
kill "$PID" 2> /dev/null

if ! wait # curl returned error
then
        echo "timeout" >&2
        rm -f /tmp/$$
        exit 1
fi

ipAdd=$(grep -Eo "$grepExpr" /tmp/$$ | uniq)
rm -f /tmp/$$

# 3  
Old 02-01-2012
Thanks...

Well the script needs to be POSIX so no sub-second sleep times. Also having a minimum time of 1 second while acceptable is not ideal, often it's done in as little as 75 ms.

It would be fine though to split the 'ipAdd' command up, after all its the url download which needs to timeout, the grep and uniq part will be lightening quick.

Code:
ipAdd=$($urlDownloaderProg "$url") 

# Do this bit later:  grep -Eo "$grepExpr" | uniq

Would that help? Or is it the assignment within the $() notation which is the problem rather than the piping?

PS. Does anyone know why both curl and wget don't seem to have a reliable "max time for whole operation"?

PPS. I know neither curl nor wget are POSIX - but they are the only bit of the script that isn't, and I've got to access the web urls somehow. Smilie
# 4  
Old 02-02-2012
It makes no sense to put a background statement in backticks, is the problem. How could it possibly set the variable until the process completes? So it either it fills the variable with a blank and puts the program in the background, or waits anyway. Neither gets you what you want.

You could replace the sleep with :, which will cause it to just loop forever until either condition breaks the loop. This will cause 100% CPU usage while it's waiting for the timeout. If you run your script with 'nice ./script.sh' that may be tolerable.

Since it needs to be POSIX you'll have to replace my math statement too.

Code:
END=`expr $SECONDS + 1`

# 5  
Old 02-02-2012
Quote:
Originally Posted by Corona688
It makes no sense to put a background statement in backticks, is the problem.
Yes indeed absolutely right.

I've spent some more time on this today and have some code as a proof of concept (I know it needs error checking added) but can you guys spot any problems/errors/bad practice with the following working code?

It has no sleep required, accepts timeouts with fractions of a second, and seems to do the job.

One problem though, the "kill -9" statement always ends up outputting a line like this (below) in the shell even with the re-direct, where am I going wrong with that?
Code:
./test: line 56:  5968 Killed   $urlDownloaderProg "$url" > $tempFileName

Code:
#!/bin/bash

# Usually responds fast ( < 200 ms)
url="http://checkip.dyndns.org"

# Usually responds slow ( > 1000 ms)
# url="http://www.dnsstuff.com/"

urlDownloaderProg="curl --silent"

tempFileName=$(mktemp --quiet "temp.XXXXXX")

$urlDownloaderProg "$url" > $tempFileName &
pid=$!

timeoutStart=$(date +%s.%N)

timeout="1.0"

finished="No"
processFinished="No"

while [ "$finished" = "No" ]; do

    ps -e | grep --quiet "$pid"
    grepRetVal=$?

    if [ "$grepRetVal" -ne "0" ]; then
        finished="Yes"
        processFinished="Yes"
    fi
    
    timeoutNow=$(date +%s.%N)

    bcExp="if ($timeoutNow - $timeoutStart > $timeout) { print 1 } else { print 0 }"

    bcRes=$(echo "$bcExp" | bc -l)

    if [ "$bcRes" -eq "1" ]; then
        finished="Yes"
    fi

done

if [ "$processFinished" = "Yes" ]; then

    tempFileContents=$(cat $tempFileName)
    grepExpr="[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+"
    ipAdd=$(echo "$tempFileContents" | grep -Eo "$grepExpr" | uniq) 
    echo $ipAdd

else
    kill -9 "$pid" > /dev/null 2>&1
fi

if [ -f "$tempFileName" ]; then rm "$tempFileName"; fi

Many thanks all.
# 6  
Old 02-02-2012
Quote:
Originally Posted by gencon
One problem though, the "kill -9" statement always ends up outputting a line like this (below) in the shell even with the re-direct, where am I going wrong with that?
Try kill without the -9.
# 7  
Old 02-03-2012
Quote:
Originally Posted by Corona688
Try kill without the -9.
Ok, so this is really strange, at least I don't get it.

I created a new script with nothing in apart from starting a process using wget to download a 100 MB file, storing the pid and then killing it. It did not matter whether I used kill or kill -9 or whether I redirected to /dev/null 2>&1 or not. The shell did not show any line of output concerning the kill at all, but the kill succeeded. But back in my 'real' script it happens every time regardless of whether I use the -9 or redirect to /dev/null - every time I get a line like this:

Code:
./getipto: line 221: 11550 Killed $urlDownloaderProg "$urlToDownload" > $tempFileName

How could it make any difference? Inside an if statement? Inside a function? I just don't understand this behaviour at all.

Any ideas anyone?

Thanks.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Using script command inside a script or emulating it

My script needs to take a folder path to the location of a C program, and run the program until the user selects exit in his C program. The script needs to create a transcript to a text file while the C program is running. Creating a transcript can be easily done with the script command but it... (2 Replies)
Discussion started by: syntax_eror
2 Replies

2. Shell Programming and Scripting

How to find whether a particular command has failed inside an sftp script?

hi, how can i know whether a command inside an sftp script has failed or not? i have a sftp expect script #!/usr/bin/expect spawn /usr/bin/sftp abc@ftp.abc.com expect "sftp>" send "cd dir\r" expect "sftp>" send "mput abc.txt\r" expect "sftp>" send "mput def.xls\r" expect "sftp>"... (5 Replies)
Discussion started by: Little
5 Replies

3. Shell Programming and Scripting

Issue with ls command inside script

Hi , DIR1 has only one file with .txt extension , trying to get the size of that file using the following script #!/bin/ksh foldr_1="/etc/DIR1" #echo "$foldr_1" sze_fdr1=$(ls -ltr foldr_1/*.txt |awk '{ print $5 }') echo "$sze_fdr1" After executing the above script getting... (1 Reply)
Discussion started by: smile689
1 Replies

4. Shell Programming and Scripting

Can i use if else inside expect command in shell script?

hii,, I am trying to automate jira. during my scripting using bash script, in the terminal i got the terminal message like this: "Configure which ports JIRA will use. JIRA requires two TCP ports that are not being used by any other applications on this machine. The HTTP port is where you... (1 Reply)
Discussion started by: nithinfluent
1 Replies

5. Shell Programming and Scripting

Using ssh command inside a script

Hi, I have a script file in server A. Inside the script file, I first have a ssh command that will connect to a remote server B. In the same script file itself, I have a sequence of commands that has to be run in server B. I am embedding these commands in the script file that I have in server A.... (2 Replies)
Discussion started by: mick_000
2 Replies

6. Shell Programming and Scripting

How to monitor a command inside shell script

Hi All, Is there any way to monitor a command inside shell script ? I have a script inside which I have a tar command which zips around 200GB data. tar zcvf $Bckp_Dir/$Box-BaseBackup-$Day.tar.gz * --exclude 'dbserver_logs/*' --exclude postmaster.pid --exclude 'pg_xlog/*' I want to... (3 Replies)
Discussion started by: sussus2326
3 Replies

7. Shell Programming and Scripting

Need help! command working ok when executed in command line, but fails when run inside a script!

Hi everyone, when executing this command in unix: echo "WM7 Fatal Alerts:", $(cat query1.txt) > a.csvIt works fine, but running this command in a shell script gives an error saying that there's a syntax error. here is content of my script: tdbsrvr$ vi hc.sh "hc.sh" 22 lines, 509... (4 Replies)
Discussion started by: 4dirk1
4 Replies

8. UNIX for Dummies Questions & Answers

Exit out of the Script Command inside a Script

I'm new to Linux. I have a bash script that invokes an executable. I'd like use the SCRIPT command inside the script and exit out of the script command after it writes to the file. Does this make sense? Below is an example of the contents of my script. #BEGIN SCRIPT script typescript... (6 Replies)
Discussion started by: jmungai
6 Replies

9. Shell Programming and Scripting

Run the command inside perl script

I have a command which will run fine in a unix command prompt. Can you tell how to interprete this command inside perl script...... The command is : perl -pe 's/(\|333\}.*)\}$/$1|1.6}/' FIA.txt This will search for the number 333 and appends 1.6 at the end of that line....... (1 Reply)
Discussion started by: vinay123
1 Replies
Login or Register to Ask a Question

Featured Tech Videos