Shell Programming and Scripting

Hi all
I´m trying to fully understand an issue I had yesterday and was just hoping you'd give me your opinion about it. I believe I fixed the problem but I still feel ignorant about some parts of the behaviour I could experience in the following scenario. Here was the situation (old simplified code version):

Code:

for FILE in $INPUT_DIR ; do
   grep -q $FILE $PROCESSED 2>&1
   if [ $? != 0 ] ; then
      echo $FILE >> $PROCESSED
   else
      continue
   fi
   SOURCE=`grep ^S $FILE | wc -l | awk '{print $1}'`
   END_RCD=`grep ^E $FILE | wc -l | awk '{print $1}'`
   if [ $SOURCE -ne $END_RCD ] ; then
      echo "Error in $FILE: "$SOURCE" source keys detected, but "$END_RCD" end keys detected.
   fi
done

The content of the INPUT_DIR contained a lot of files, and it reported the above error on one of them (SOURCE = 1, END_RCD = 0). However the file was ok as far as I could see. I quickly ensured my commands were correct and even re-ran the script on the whole file list to get a final ok as an output, so this looked weird at first sight.

After some time of investigation I finally found that I had a bunch of n processes running in the background calling that bit of code. Despite of this I originally supposed this script would only be running once at the same time, so the files processed from my process(n) falling into INPUT_DIR would be updating the PROCESSED file so they don't get picked up by the next calls.

Unfortunately for my theory, the truth was that the script got called several times at the same time (from different sessions), processing in parallel, this with a PROCESSED file that wasn't unique (you begin to see what it can look like here). So to summarize, the above script reported an error when running on the process (n) for a file created by the process(n-1).

Now the above situation should still be ok as long as a file is opened once a time, it's just that this file gets checked by the wrong process but the result remains the same. So what I believe is that the file on which it reported the error was opened in two sessions at the same time.

I have removed the PROCESSED file and now run this script once after all processes are completed so I don't get any issue anymore. The only thing I still don't understand is why END_RCD gave 0. Since two sessions have different variable buffers, I don't see how a potential clash on a file would produce such an error. If anybody here has a credible explanation, I would be very happy to know about it.

Many thanks,
Yann