Concurrency in ksh


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Concurrency in ksh
# 1  
Old 01-16-2012
[solved] Concurrency in ksh

Hi, first I have a doubt in order to run a single script several times at same time I used this:

Code:
for i in {1..n}; do ./script.sh & done

Where n is the times will run the program at the same time.

Is it this correct?

Second, I have a script that read a text file and modify it and I want to simulate a semaphore in order to have the same script running several times at the same time and just one can edit and when it finish the second will edit and so on..., but in ksh I can't use lock file... So on internet I look for another way and I found that it can be do it with mkdir cos is an atomic operation, so I implemented on my code.

Code:
#!/usr/bin/ksh
 
 
lock_wait(){
 
path="/home/yadis/tests"
rmds=$((RANDOM%10))
 
    while true; do
       if mkdir "${path}.lock.d"; then
         break;
         fi
      sleep $rmds
   done
}
 
remove_lock(){
path="/home/yadis/tests"
rmdir "${path}.lock.d"
}
 
leer(){
 
   fecha=`date +%d`"/"`date +%m`"/"`date +%Y`"_"`date +%H`:`date +%M`:`date +%S`
lock_wait
 
   num=`cat file.txt`
   print $num "file.txt" $fecha 
   print $num "file.txt" $fecha >> filelog
 
num=$(($num+1))
print $num > file.txt
remove_lock
 
if [[ num -ge 100 ]]; then
exit 1
else
 leer
fi
 
}
 
fecha=`date +%d`"/"`date +%m`"/"`date +%Y`"_"`date +%H`:`date +%M`:`date +%S`
 
print "script.sh" $fecha
 
leer

But it doesn't work, and the sentence that is in red I don't understand very well.
Cos Always I run the program never enter to the sleeep, so I think the if always is true and then it's breaks the loop, but I'm not sure.
And also for some reason it doesn't create the directory... I don't know why

Any suggestions?

Thansk ^^

Last edited by Sui; 01-20-2012 at 05:18 PM..
# 2  
Old 01-16-2012
Yes, your for loop will start n concurrent executions of the same script.

The statement you have in red attempts to make a directory. If it is successful, the if should evaluate to true and the loop should be exited. I ran your lock wait function and it works for me.

As an alternative example, here is a stripped down version of what I use. Similar, but different enough to give you a second perspective. You can try running it twice from the command line to see if it is able to create a lock and hold it for a while.

Code:
# pass file to lock as $1
function lock_it
{
    typeset lockname=$1.lock

    while ! mkdir $lockname 2>/dev/null    # if we can make it, we own the lock
    do
        echo "$$ waiting for lock"      # didn't get it, wait and try again
        sleep 3
   done

    touch $lockname/PID$$       # mark it with our pid (humans can easily tell which process owns the lock this way)
}

# pass file to unlock as $1
function unlock_it
{
    typeset lockname=$1.lock
    if [[ -d $lockname ]]      # ensure it exists
    then
        if [[ -f $lockname/PID$$ ]]    # ensure this process owns it
        then
            rm -fr $lockname
        else
            echo "unable to unlock $w: process $$ does not own the lock"
        fi
    else
        echo "unable to unlock $1: lock does not exist"
    fi
}


# simple test case
lock_it /tmp/test
echo "$$ got it"
sleep 30
echo "$$ releasing it"
unlock_it /tmp/test

In the code that I use in production I allow for a max amount of time before the lock function gives up and returns an error. It also supports a no-wait option that allows for a non-blocking approach.
This User Gave Thanks to agama For This Post:
# 3  
Old 01-17-2012
Quote:
The statement you have in red attempts to make a directory. If it is successful, the if should evaluate to true and the loop should be exited. I ran your lock wait function and it works for me.
Then my pc hates me XDD, ok no xD

But then if is creating a directory and it is successful the loop will be over and never enter to sleep??? or it is the opposite??

But it's very strange that it is not creating any directory for me....

if I put the next:

Code:
while true; do
    mkdir "${path}.lock.d"
   if [[ -d "${path}.lock.d" ]]; then          
 break;         
 fi       
 sleep $rmds    
done

is it still the same condition?
I will try this and then I will edit xD

----edit
Ok I tried it, now It's creating the directory, but I think never will enter to the sleep xDDD

Also I have another question, seeing your example, sometimes I tried to do a statment like this one:
Code:
 while ! mkdir $lockname 2>/dev/null

But when I run the program it says that !: not found, why is that?

Last edited by Sui; 01-17-2012 at 12:28 AM..
# 4  
Old 01-17-2012
Quote:
Originally Posted by Sui
Then my pc hates me XDD, ok no xD

But then if is creating a directory and it is successful the loop will be over and never enter to sleep??? or it is the opposite??
If it is successful on the first try it should not enter the loop and once it is successful it should exit the loop.

Quote:

Ok I tried it, now It's creating the directory, but I think never will enter to the sleep xDDD
Great ! Try running two of them at the same time. The first should create the directory and the second should block until the first frees the lock.

Quote:

Also I have another question, seeing your example, sometimes I tried to do a statment like this one:
Code:
 while ! mkdir $lockname 2>/dev/null

But when I run the program it says that !: not found, why is that?
That sounds like an old version of the shell. What is the output of the command
Code:
ksh --version

I think you were using kshell. If you have an older version, before 2002/02 you might want to install a new one.
# 5  
Old 01-17-2012
yes I'm using ksh, I think It will be a little dificult for me cos it's has to work in that old version..... e_e but well I'll don't give up until I tried everything xD

Quote:
Great ! Try running two of them at the same time. The first should create the directory and the second should block until the first frees the lock.
I did that with five... so I think it isn't working.... damn old version ksh ¬¬

I was reading something about using a "lockfile" but no the lockfile that needs to install a package in order to use it, it is a hand made lock file.
So I rework on my code and try something:

Code:
#!/usr/bin/ksh
 
#!/usr/bin/ksh
max=$1
fecha=`date +%d`"/"`date +%m`"/"`date +%Y`"_"`date +%H`:`date +%M`:`date +%S`
print "process3.sh" $fecha
print "process3.sh" $fecha >>file.log
lock=0
until [[ $num -ge $max ]]; do
  fecha=`date +%d`"/"`date +%m`"/"`date +%Y`"_"`date +%H`:`date +%M`:`date +%S`
alea=$((RANDOM%3))
if test -f archivo.lock
then
    lock=1
    fi
    while [[ $lock -eq 1 ]] 
  do
    if test -f archivo.lock
    then
      lock=1
      print sleeping
      sleep $alea
    else
      lock=0
  fi
done
touch archivo.lock
  num=`cat archivo.txt`
  num=$(($num+1))
  print $num > archivo.txt
rm -f archivo.lock 
   print $num "archivo.txt" $fecha >>file.log
   print $num "archivo.txt" $fecha
done

About what I read I guess I need to have this: while exist the lock file the process needs to wait, create the lockfile, do the stuff of the main script, delete the lockfile and repite the process I think....

at first sight i think is working... but I think never enter to the sleep, I never see the "sleeping"

But now I have a doubt, I really don't know if this:

Code:
for i in {1..n}; do script.sh & done

is it working well?

cos when I put:

Code:
ps -ef | grep script

just appears 1... I think it has to appears the n process that I run right?

is it another way to run a script n times at same time?

Last edited by Sui; 01-18-2012 at 05:08 PM..
# 6  
Old 01-18-2012
The lock file code in your last post is bound to have problems. If two instances of your script both notice that the lock file is missing at the same time, they will both touch the lock file and go on. Not that it matters in this case, since touch will always work, you don't check the status of the touch. What ever command you use to 'create' the lock file must allow only one process to be successful and the script must test the result and loop back to try again when it detects a failure.

The reason mkdir was used is because only one process will successfully create the directory if two attempt the operation at exactly the same time. Also, the return value of mkdir is also being checked and if it didn't succeed the script assumes it does not own the lock.

Assuming the seq command is installed on your machine, you could run this to start 5 processes:

Code:
for x in `seq 1 5`
do
    shell-script >/tmp/log.$x 2>&1 &
done

Here is a version of my original code in what I think should execute under a very old version of ksh. It starts 5 copies of itself and each waits to get the lock reporting when it does and does on the tty. Cut and paste it into its own file and run it (just once) to see what happens. If this doesn't work (it does on my system) then I'm not sure what to do to help you along.
Code:

lock_it()
{
    lockname=$1.lock

    while  true
    do
        if mkdir $lockname 2>/dev/null
        then
            touch $lockname/PID$$       # mark it with our pid
            return 0
        fi

        echo "$$ ($2) waiting for lock"
        sleep 3
   done

}

unlock_it()
{
    lockname=$1.lock
    if [ -d $lockname ]
    then
        if [ -f $lockname/PID$$ ]
        then
            rm -fr $lockname
            return 0
        else
            echo "$$ unable to unlock $w: process $$ does not own the lock"
        fi
    else
        echo "$$ unable to unlock $1: lock does not exist"
    fi

    return 1
}

if [ -z $1 ]
then
    for x in `seq 1 5`
    do
        echo "starting process $x"
        $0 $x &
    done

    ps -elf|grep "$0"
    wait
    exit
fi

lock_it /tmp/test $1
echo "$$ ($1) got the lock; sleeping 30s"
sleep 30
echo "$$ ($1) releasing lock"
unlock_it /tmp/test
echo "$$ ($1) is exiting"

exit

This User Gave Thanks to agama For This Post:
# 7  
Old 01-19-2012
yeah, that works!!! At less the functions.

the seq isn't working on my ksh version and I can't installed, but I make this:

Code:
i=0
while [[ $i -lt $1 ]]; do
    ./script.sh $2 &
  i=$(($i+1))
   done

And it's working ^^

I tried to do something similar to your program where is the seq, but i think it didn't work, it never enter to the if, but the functions are working.

now I have a doubt, when I run my script, one process got the lock, the others just are waiting and the end they got the lock and don't respect my condition to stop at X number.

do you know why?

Code:
#!/usr/bin/ksh
max=$1
lock_it()
{
    lockname=dir.lock
while  true
    do
    if mkdir $lockname 2>/dev/null
    then
    touch $lockname/PID$$       # mark it with our pid
    return 0
    fi
    echo "$$ waiting for lock"
    sleep 3 
     done
}
unlock_it()
{
    lockname=dir.lock
     if [[ -d $lockname ]]
      then
    if [[ -f $lockname/PID$$ ]]
    then
   rm -fr $lockname
    return 0
    else
  echo "$$ unable to unlock $w: process $$ does not own the lock"
   fi
    else
    echo "$$ unable to unlock $1: lock does not exist"
   fi
 return 1
}

until [[ $num -ge $max ]]; do
  fecha=`date +%d`"/"`date +%m`"/"`date +%Y`"_"`date +%H`:`date +%M`:`date +%S`
alea=$((RANDOM%3))
lock_it
print "$$ got the lock" 
 print "$$ got the lock" >> file.log
 sleep $alea 
 print "$$ releasing the lock"
  num=`cat archivo.txt`
  num=$(($num+1))
  print $num > archivo.txt
  unlock_it 
  print "$$ exiting" 
  print $num "archivo.txt" $fecha  >>file.log
   print $num "archivo.txt $$" $fecha
done

Login or Register to Ask a Question

Previous Thread | Next Thread

7 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Writing a REST server to run on FreeBSD -- how to structure for concurrency?

Hi All, I want to write a domain specific REST/HTTP server to run on FreeBSD. I have control over both the server side and the primary client intended to consume the service. My question is: how do you think it should be designed to support multiple connections, given: It will run on a... (6 Replies)
Discussion started by: LittleCookieMon
6 Replies

2. UNIX for Dummies Questions & Answers

Open-source projects to learn concurrency-managed network programming in Unix?

Hi, I am a mid-career programmer with extensive experience in object-oriented design and development in C, C++, and C#. I've written a number of multi-threaded server applications and background services, although my grasp of networking protocols is a bit weak: my current job drifted away from... (2 Replies)
Discussion started by: TheTaoOfPhil
2 Replies

3. Shell Programming and Scripting

different behaviour for ksh and ksh -x

I'm getting different behaviour when executing below script in debug option. $ cat ss.ksh ff=$(pwd) echo " ff : $ff" $ ksh ss.ksh ff : /tmp $ ksh -x ss.ksh + + pwd ff= + echo ff : ff : I was getting this behaviour in my actuall script i'm able to reproduce this in simple script... (4 Replies)
Discussion started by: luckybalaji
4 Replies

4. UNIX for Dummies Questions & Answers

Difference Between executing llike ./myscript.ksh and . ./myscript.ksh

Hi , What is the diffence between executing the script like ./myscript.ksh . ./myscript.ksh I have found 2 difference but could not find the reason 1. If i export a variable in myscript.ksh and execute it like . ./myscript.ksh the i can access the other scripts that are present in... (5 Replies)
Discussion started by: max_hammer
5 Replies

5. Shell Programming and Scripting

KSH script to run other ksh scripts and output it to a file and/or email

Hi I am new to this Scripting process and would like to know How can i write a ksh script that will call other ksh scripts and write the output to a file and/or email. For example ------- Script ABC ------- a.ksh b.ksh c.ksh I need to call all three scripts execute them and... (2 Replies)
Discussion started by: pacifican
2 Replies

6. Shell Programming and Scripting

import var and function from ksh script to another ksh script

Ih all, i have multiples ksh scripts for crontab's unix jobs they all have same variables declarations and some similar functions i would have a only single script file to declare my variables, like: var1= "aaa" var2= "bbb" var3= "ccc" ... function ab { ...} function bc { ... }... (2 Replies)
Discussion started by: wolfhurt
2 Replies

7. UNIX for Advanced & Expert Users

concurrency issue while Accessing Mail Box from shell script

Hi Bros, I am in a serious trouble with concurrency issue while using mailx. I have a shell script which reads all the emails of a unix user account and create a copy (another mbox for processing and archive purpose). This script works fine for 99.99% of the time but sometime it start creating... (2 Replies)
Discussion started by: Sumit_Fundoo
2 Replies
Login or Register to Ask a Question