Shell Programming and Scripting

Hi,

I need to send a file from one server to at a set time interval, once the file has been completely sent, it is then processed on the receiving server. Is there a way that i can send a file and know that the complete file has been sent and that there have been no errors in the transmission of the file?

I was thinking of sending the file via scp/ssh and then once the main file has been sent, send another flag file to confirm that the first file has been completely received.

Is there a better way of doing this, as i am thinking that there could be a problem in transferring of the flag file, so the first main file would have transferred, but the flag file would not have, and therefore the file would not be processed.

Any ideas?

Thanks

I would suggest write a script on destination server which will just touch a <<filename>>.done file and invoke it from source system once main file transmission is done. This will ensure file has arrived successfully on destination folder and if it fails in between .done job wont get executed.

That is one approach, but if there was an incomplete transfer of the file i.e. if the contents were not complete, then the .done file would still be created which would be a problem. Or when using scp is there no chance of an incomplete transfer?

If files are critical then one cant trust scp or ssh. You have to go for tools like Connect Direct etc.
SCP do return $? more than 0 when there is a failuer in copy action.

brunlea,
You don't mention how or when the file is processed on the receiving server. Does the file get processed as soon as it is transferred (i.e. via some file watcher) or does a script run at a certain time to process it? I have used Cyberfusion and Connect Direct to transfer files and they reliably tell you when there is a transfer failure or file is incomplete. Don't have much experience with scp/ssh capturing all errors (I'm sure someone else can weigh in on this), however, something you could do is compare chksum's but this would need to run after file has been transferred (again, that depends on my question on how/when it's processed).

Yet another option is to put a trailer record at the end of the file you are transferring that contains the total number of records on the file. Then the script on the destination server could validate that the total number of records on the file matches the total on the trailer record before processing it.

The file gets processed as soon as it is received, this is why i need to ensure that the full file has been received. The file processer can check first for a flag file etc.

No other tools are available to be installed so scp/ssh is the only option available for me.
The trailer option may be viable. Never done that before, how does that work?

Below is just one example of how you can define a trailer record at the end of the file. The trailer record has a record type (1st char of rec) of '3' and is followed by the total count (000004) of detail records (those with a record type of '2' in this example). The process that reads this file after it's transferred should count the number of actual detail records on the file and compare that number to the total count on the trailer record. If the count is different or if no trailer record is found, then that means that the transfer or file is incomplete. This file also contains a header record with a date which the process can also validate so an older or unexpected file is not processed. Surely there can be many variations to this.

Code:

 
cat file.txt
1 20130430
2 Detail Rec 1
2 Detail Rec 2
2 Detail Rec 3
2 Detail Rec etc
3 000004