Shell Programming and Scripting

Hello,

I have a shell script that runs every minute to process incoming files delivered externally via SFTP to a directory. Basically the script works and processes the files however I get an error when a new file is delivered into the directory.

Please see my script below. A new file is delivered every day via SFTP from an external source which the script processes into the correct naming convention and it is then imported into our database using curl. We get a new file at 06:05 every day and I get a cron error message delivered to my mail box every morning. However if I then go and change the script and then test it using the same file with cron it will process correctly. However when I come in in the morning I will find that it has failed again - very frustrating!

I have no idea why this is! If someone can suggest anything that may remedy please let me know. I have also included a copy of the error message I get from Cron.

#!/bin/bash
TRANSDATA=/home/tsbftp/transaction_data
WORKING=/home/tsbftp/working
FILES=$(ls $TRANSDATA)
IMPORT=/home/arch3ftp/dfn-dev.arch3systems.com/html/import
for file in $FILES
do

#change owner of file to root - change permissions for root to full access
echo "$file" > /tmp/formatfile.nfo
chown root $TRANSDATA/"$file"
chmod 777 $TRANSDATA/"$file"

#Strip out the account number and place into a variable
AN=$(grep ^00 $TRANSDATA/"$file" | cut -c 3-18)

#set current date and time for file name convention and place into a variable
CUDA=$(date +%Y%m%d%k%M%S)
#Rename Current File with Required Format
mv $TRANSDATA/"$file" $TRANSDATA/$AN"_"$CUDA.a3i
sleep 2
mv $TRANSDATA/$AN"_"$CUDA.a3i $WORKING
echo $? >> /tmp/formatfile.nfo

cp $WORKING/$AN"_"$CUDA.a3i $IMPORT
curl -s http://dfn-dev.arch3systems.com/batches/import/$AN/$CUDA
done
exit 0

As you can see I've added in sleep 2 etc to try and remedy the error I get and tried renaming the file in place then moving etc. Please see error message below from Cron.

/etc/cron.d/format_file.sh:

mv: when moving multiple files, last argument must be a directory Try `mv --help' for more information.
cp: cannot stat `/home/tsbftp/working/1539590548547000_20080305': No such file or directory

I always get the mv error, however if I change the script to do the rename a different way it will work when I test it with cron! However sure enough it will fail when a new file comes in the next day!

Because when you test, the sftp is completed and nothing is locking the file letting you rename/move with no problems.

You probably need to change the way the sftp works - meaning, send a second file (either a 0 byte file or one with the checksum of the first file in it), check that the second file is there (due to it's small size, you can sleep 2 seconds and have no issue opening it to grab the checksum info).

Steps:
1. sftp sending original file
2. Your cron job kicks off - sftp may still be sending file
3. Cron job waits for 2nd file.
4. sftp finishes sending first file.
5. sftp sends second file.
6. cron job finds second file - 'knows' it's safe to manipulate first file
7. If you decide to send checksum info - check info sent versus what your system shows as checksum - should be the same.

If that can't be done (you don't control what sftp is sending), then you can list the original file - wait until it stops getting bigger. Or, look for the sftp process, wait until it is gone. But the checksum is a great way of insuring your file is the same as original.

thanks for the reply I understand what u r saying but surely it would process eventualy when the file is complete? as it runs every minute. The file will not process even if i run it manually on the new file that has come in even after waiting a couple of minutes. It's only after i change the script slightly will the file process corectly until a new file appears next day.

I will try what u have suggested as I have run out of ideas. Thanks again.

Richo