Shell Programming and Scripting

my user copy large files, and it's take 10min for file to be copied to the server (/tmp/user/ files/), if in the meantime start my scheduled script, then it will copy a part of some file to server1

my idea is to check the file size twice in a short period (1-2 seconds) of time, then compare, if the size is different then script transfer1.sh will copy it the next time wen is started from cron,

If anyone has an idea how to do this or any other suggestions?

PS another thing, files name will be random, but every file have "TEST" in name.
when my script run there will be about 5 files to transfer, and this control is just for case if one of the files is in copy process, but other files must be transfered to server1

30 * * * * /myscript/transfer1.sh

Code:

#! transfer1.sh  script

cd /tmp/user/files/
for f in `ls *TEST1*` 
#! here i need check if some file is in copy process
do
scp -qB $f user1@server1:/tmp/end/file/$f
rm -rf /tmp/files/$f
done

example, folder contains
/tmp/user/ files/
TEST1.out
TEST2.out
TEST3.out
TEST4.out (example this file is still copying and will bee skiped with transfer to server1)

after transfer delete all files except TEST4.out, that file will bee copyied next time when script is started

Bad idea -- how long is long enough? Best idea: write under another name like same.temp and then rename when done. Second choice: use fuser to see if it is open, but even if not open, file may be incomplete due to a failed writer, unless you have a way to validate it, such as a trailer record.

PS: Comment s/b after the do line!

Instead of checking the file size you should check if the file is open. You can use the fuser utility to do that:

Code:

f=/tmp/x.x
test ! -f $f && echo file not found && exit
c=`fuser -f $f 2>&1 | wc -w`
c=`echo $c`
case $c in
1) echo file is not open;;
*) echo file is open;;
esac

Tested with bash on OpenBSD

---------- Post updated at 12:57 PM ---------- Previous update was at 12:54 PM ----------

Agreed. Copy/rename is the fool proof way to avoid problems.

Well, some people send additional short ack files, perhaps with a line or byte count or checksum, but that is excess complication if you can rename.

Code:

if [ -f "$f" -a "$( fuser "$f" 2>/dev/null )" = "" ]
then
 # file is not open
fi

Since fuser puts the pids on stdout and the rest (path name, colon, open type indicators) on stderr, you can just test for pids or you can ps -fp the pids to show who is on the file!

another thing, files name will be random, but every file have "TEST" in name.

when my script run there will be about 5 files to transfer, and this control is just for case if one of the files is in copy process, but other files must be transfered to server1

So, did you pick a robust method of dealing with partally completed files?