AIX

I need to change how a posting procedure currently works in order to improve load balancing but I am hitting a potential file contention problem that I was wondering if someone here could assist me with...

In a directory called FilePool I would have a bunch of files that are constantly coming in as follows:
File0001.GO
File0001.GOA
File0001.GOB
File0001.GOC
File0002.GO
File0002.GOA
File0002.GOB
File0002.GOC

The *.GO file for each batch of files is only written after the entire batch has been written, so the problem isn't that what is pulling these batches may conflict with what is writing them, but rather:

There are four processes that are each looking for the presence of a *.GO file in the FilePool directory. If one of these processes finds one, it takes the entire batch and moves it into that process' directory to process the batch.

These processes work quickly and it could be that two processes try to grab File0001.GO at the same time. Is there a way to
1 - have the first process lock File0001.GO if it isn't already locked and then move the whole batch
2 - have the process check if there is any other pid attached to that file first and then move it if there is none
3 - any other way that you would ensure that the file contention in this situation is handled?

Thank you for any assistance you can provide.

for #1, can you temporarily rename the file?
for #2, you can use

Code:

ps -eo pid,args|grep "*.GO$"

, or

Code:

ps -auxww|grep...

This will give you a list of process id's where the command args contain the filenames. 'man ps' for more info -- it's flavor-dependent, i.e., on solaris you might need /usr/ucb/ps instead (see this thread https://www.unix.com/shell-programmin...cates-o-p.html)

Can you have your processes write to an index file? Each proc could then look in this file for exclusions (i.e., filename in file, omit it from search,) write to the index when it claims ownership, and remove the entry when complete.

I'm going to try your renaming suggestion first - I've created a script in each of the two process directories to take the first *.GO file and mv it into the process folder. I am going to use at or cron to call these two scripts simultaneously and see what happens.

Presumably if you check the return value of mv for the rename then the one which attempts to run second will have failed and you can safely exit. Interestingly that avoids the race condition of trying to identically-named files at the same time.