Shell Programming and Scripting

I'm changing my mindset from a few big processes moving data from a few sources under an external, dependency-based scheduler to multiple processes moving data from many sources run by each client cron and possibly interfering with each other. It has the benefits of more granular code but I'm worried about tripping myself up. The incoming data will be delivered (atomically) into a known directory, but we cannot predict when so we have the script to process & remove them run every few minutes. The problem is that it could run a long time if we have a huge volume of data. I am therefore looking at locking mechanisms.

The flock manual page refers to wrapping up commands in parenthesis , redirecting a file descriptor to a file, something like this:-

Code:

(
flock -s 200
    # ... commands executed under lock ...
) 200>/var/lock/mylockfile

It has been suggested elsewhere that one could use braces {&} to acheive the same and avoid spawning another process. Sadly, both of these would make my code a little ugly because I want an exclusive lock in force for almost the whole script. It's not massive (only about 600 lines) but it adds an indentation I'd rather not have and I'm torn because I want to be good about indenting code too.

Additionally, whist debugging, I don't want to have to worry about where the end of the block of code is and generating very confusing syntax errors.

I have discovered that this seems to work just as well and doesn't leave me feeling I should be indenting the code wrapped in parenthesis/braces or worried when looking to grow/debug:-

Code:

#define any functions first

# Set/test lock
exec 200>/var/lock/mylockfile
flock -w 1 -x 200 || exit                 # Do not run concurrently

# My main script starts here
sleep 10                                  # Just for illustration, honest!
# My main script ends here

flock -u /var/lock/mylockfile             # Is this necessary?

The script is simply run by cron and I don't want more than one at a time, hence the exclusive lock. There would never need to be output written to the lock file.

So, my questions:-



Thanks, in advance,
Robin

Hmm. The idea of flock is to wrap a single (usually) command within the duration of the lock. If the command fails midway, then the caller still calls unlock.

A six hundred line script is doing an awful lot of stuff for the duration of the lock - just a point. There must be a large number of possible points of failure - what happens on a busy system? For your structure to work, you want there to be a single thread of active execution. And you are using the lock as a traffic signal.

So is your lock request at a suitably granular level? I do not know. I'm guessing no.
But you do have to unlock no matter. You probably need a simple master gate minding script to take care of minding the lock, and calling your 600 line beastie.

I think the _exit(2) of the shell is sufficient, no need to explicitly unlock.

However... will you ever leave background processes ? The descriptor would be passed to them, and, depending on the exact locking mechanics the lockf(1) uses, the lock might live on until those processes exit too.

Do you ever delete the lockfile ? There's a possibility of race if so.

Would an utility wrapper like

Code:

exec 3> thelock || exit
lockf -w1 -x3 || exit
the_real_job 3>&- &
wait

simplify/beautify things? Note fd 3 is closed instead of passed to the child.

Juha

Meh, I'm an idiot...

Code:

&
wait

serves no purpose

Thanks jim mcnamara,

Yes, it is a bit of a beastie but I do want to ensure there is only one of the whole thing running. It's collating data from multiple sources and there could be multiple inputs per source. The script itself runs fine, but could be slow for volumes of data.

Talk about me missing the blindingly obvious solution to stop me worrying. That is indeed a very sensible solution.


Thanks also to Juha Nurmela, good stuff, but sadly these are all C functions and I'm just in the shell. Sorry, I should have said.

If it matters, it's bash on CentOS v6.



Kind regards,
Robin

Surely the shell also uses those system calls.

There are oddities in file locking, see the manpages if you care. One oddity is that if you open a file, lock it, open it again and close that latter fd, the lock is lost.

I got flock and lockf mixed up above, sorry. Here on FreeBSD, and the "scriptable" locker is called lockf.

Juha

Can you use inotify on linux systems ?
If files are moved as you say atomically into known (existing?) directory, inotifywait should catch the exact moment using moved_to.

Perhaps even eliminating need for locks and cron jobs.

A script from screen perhaps, when moved_to action stops for N <insert your time> , do your magic or something like that.

It should simplify the procedure, but will be limited to linux systems.

Hope that helps
Best regards
Peasant.