Shell Programming and Scripting

As another wild guess here, it sounds to me like ningy is starting to copy files while they are being written and then removes them at the source (while they are still being written). I think the code needs to be modified to be sure the file is complete before the move starts.

Thanks for the inputs everyone.

Yes, I was doing :

Code:

cp -rp source destination
ln -s  dest source
rm -rf source

But failed while some new files getting written to.

I have just started exploring find command with -newer option to check if new files are added by creating a file just before copy and checking before removing source. Not sure if thats the best thing to do but a start atleast

Yes much better.
The goal is to make a check as close as possible.
I would even go for

Code:

find -type f -cmin +1 ... cp ...

if you have GNU find.
The maybe best approach is like your suggestion, but twice:

Code:

cd /path/to/source || exit
# create an empty file for a time comparison
 > findstart
find . -type f -print |
while IFS= read -r file
do
 # don't copy file if newer
 find "$file" -newer findstart | grep . >/dev/null && continue
 # another reference file
  > copystart
 cp -rp "$file" /path/to/destination/ || continue
 # don't delete file if newer
 find "$file" -newer copystart | grep . >/dev/null && continue
 rm -f "$file"
done
rm -f findstart copystart

If I understand what you're saying, it won't solve your problem. You don't need to know if a file is new before you remove it; you need to know that a file is complete before you start copying it. You can only do that by having the server provide some indication that the data in the new file is complete. The client can't reliably know that the source file on the server is complete unless the server provides some unambiguous way to determine that.

What MadeInGermany recently proposed is a big step in the right direction, but there is still no guarantee that the process loading the file being copied will not have been sleeping or "swapped out" while the copy to the client was being processed.

You could easily eliminate all of your headaches if you had a directory on the same filesystem as "source" that was dedicated to files in flight. Since it knows when it's done with a file, the script writing the file should be in charge of mv'ing from the in-flight directory to "source". This way, every file in "source" is guaranteed to be complete.

In my opinion, this is the simplest and most robust solution. Anything else will be either more complicated or less dependable or both.

Regards,
Alister