UNIX for Dummies Questions & Answers

Hi there,

i've a question about cronjobs. I'm creating a concept for a centralized logging repository using log4j/log4net. Sadly the appenders I want to use (fileappenders) aren't telegram based but need a permanent stream to the repository. Because I can not assure this I want to log these logs local on the machines and copy them via cronjob.

Since these log files can become really huge, and I need an update of those logs all 30-60minutes on the repository, I'm looking for a possibility to only copy the new entries which are not yet on the repository. Because if I would always copy the complete file, there would be an extreme overhead of network traffic.

Is there any possibility to solve this?

Thanks in advance for your help.


Greetings and have a nice day,

collatz

rsync will update files very efficiently, we use it in cronjobs all the time to maintain copies of large files.

Over a network, rsync will transfer only the parts of a file that have actually changed. A giant log file only actually transfers the newest line. The algorithm that accomplishes this uses a "rolling checksum" and is well described at The Rsync Algorithm

Thank you for your answers. I just tried to synchronize logs with "SyncToy". I hope this uses not the same principle then rsync. If I try to synchronize a log and at the same time there is a new log entry written into the source file, the sync process fails.

Does this problem also exist with rsync?



greetings,

collatz

There are no psychic backup programs. If you can't stop the file from being modified during transfer, you can't guarantee any backup process.

If rsync detects a file changing during transfer, the checksum will fail and it will retry, once. If it happens twice in a row it will give up.

Hmmm... is there a possibility to set the log file write-protected automatically for the duration of the sync? I know there is the chance that log entries cannot been written into the file, but the probability should be really low...

If you need to "freeze" data for a backup, you can use a snapshot. See 2004_12_24.html snapshot