Visit Our UNIX and Linux User Community


Is this possible with cronjobs?


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Is this possible with cronjobs?
# 1  
Old 07-23-2009
Is this possible with cronjobs?

Hi there,

i've a question about cronjobs. I'm creating a concept for a centralized logging repository using log4j/log4net. Sadly the appenders I want to use (fileappenders) aren't telegram based but need a permanent stream to the repository. Because I can not assure this I want to log these logs local on the machines and copy them via cronjob.

Since these log files can become really huge, and I need an update of those logs all 30-60minutes on the repository, I'm looking for a possibility to only copy the new entries which are not yet on the repository. Because if I would always copy the complete file, there would be an extreme overhead of network traffic.

Is there any possibility to solve this?

Thanks in advance for your help.


Greetings and have a nice day,

collatz
# 2  
Old 07-23-2009
rsync will update files very efficiently, we use it in cronjobs all the time to maintain copies of large files.
# 3  
Old 07-23-2009
Over a network, rsync will transfer only the parts of a file that have actually changed. A giant log file only actually transfers the newest line. The algorithm that accomplishes this uses a "rolling checksum" and is well described at The Rsync Algorithm
# 4  
Old 07-23-2009
Thank you for your answers. I just tried to synchronize logs with "SyncToy". I hope this uses not the same principle then rsync. If I try to synchronize a log and at the same time there is a new log entry written into the source file, the sync process fails.

Does this problem also exist with rsync?



greetings,

collatz
# 5  
Old 07-23-2009
There are no psychic backup programs. If you can't stop the file from being modified during transfer, you can't guarantee any backup process.

If rsync detects a file changing during transfer, the checksum will fail and it will retry, once. If it happens twice in a row it will give up.
# 6  
Old 07-24-2009
Hmmm... is there a possibility to set the log file write-protected automatically for the duration of the sync? I know there is the chance that log entries cannot been written into the file, but the probability should be really low...
# 7  
Old 07-27-2009
If you need to "freeze" data for a backup, you can use a snapshot. See 2004_12_24.html snapshot
 

Previous Thread | Next Thread
Test Your Knowledge in Computers #828
Difficulty: Medium
JSON and ECMAScript 2018 both allows the Unicode line terminators U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR to appear unescaped in quoted strings.
True or False?

10 More Discussions You Might Find Interesting

1. Solaris

How to find all Cronjobs?

Hey Guys, i've got a big issue... I've to find all running scripts in all crontabs. Is there a possibility to display all crontabs of each user? What i've already tried? The following script: for user in $(cut -f1 -d: /etc/passwd); do crontab -l $user; done I'm already root but i didn't... (3 Replies)
Discussion started by: Marcusg562
3 Replies

2. UNIX for Advanced & Expert Users

cronjobs orphan processes

Our cron job stats its started on Oct20 % ps -ef | grep cron root 1442044 1 0 Oct 20 - 25:23 /usr/sbin/cron All the below jobs aixmf,aixgh are triggered from cron only. user pid ppid date time cmd gaix 1581282 1 35 16:33:01 - 20:56... (1 Reply)
Discussion started by: karnan
1 Replies

3. Shell Programming and Scripting

Setting up cronjobs

hello all, I have a shell script and I need to schedule it in crontab, I have the next line: 06 16 * * 1,2,3,4,5 /usr/bin/ksh /path/path/name.sh > /path/path/name.log first, I scheduled from Monday to Friday but it doesn't run, the log file is empty.. any idea why is causing this?... (14 Replies)
Discussion started by: Geller
14 Replies

4. Shell Programming and Scripting

cronjobs stopped working

Hello people, I had these cronjobs scheduled in some Unix boxes which were running fine until yesterday.But then the password was changed for that user id and then the jobs stopped working. As far as i know cron jobs run from super user. I am completely lost over here now. Thanks. (2 Replies)
Discussion started by: King Nothing
2 Replies

5. Solaris

cronjobs not running.

hi friends, how to check if the cronjobs is not running and how to make it run again. (1 Reply)
Discussion started by: cromohawk
1 Replies

6. Linux

Cronjobs stopped working

Hi All, I am user of a Linux machine and I have approximatly 15 cronjobs scheduled in my crontab. Yesterday my administrator made LDAP active on my userid and all the things are doing fine after that. But all cronjobs for my user id stored in my crontab have stopped working after that. Could... (1 Reply)
Discussion started by: bisla.yogender
1 Replies

7. Shell Programming and Scripting

Setting cronjobs...

Hi, We have 4 jobs to be run every month on different times - * a daily job runs once in 2 days at 3PM *a weekly runs every thursday at 3PM * a monthly runs last day of month either 30 or 31st at 3PM * 4th job runs on 3rd of every month at 3Pm How can I set the crontab for these 4 jobs... (4 Replies)
Discussion started by: krworks
4 Replies

8. UNIX for Advanced & Expert Users

Delete Duplicate Cronjobs

I set up same cronjobs in two different users to generate messages at 5:30 AM Not Its generating duplicate messages. I want to delete the cron entries set up in the first user, but I am unable to view the entries in that user. I tried to find the process Id, but its not showing any id Could... (2 Replies)
Discussion started by: nskworld
2 Replies

9. AIX

Cronjobs

We recently upgrade from AIX 4.3.3 to AIX 5.3, We noticed that some cronjobs that run for our programmers did not fire off this morning. You can crontab -l and -e and see the jobs. Did AIX 5.3 change something? Thanks Mike (1 Reply)
Discussion started by: mcastill66
1 Replies

10. UNIX for Dummies Questions & Answers

cronjobs

hi How can I add a cronjob to the crontab file? to execute a shel script named testScript.sh every day at 00:00. Thanks (3 Replies)
Discussion started by: tamer
3 Replies

Featured Tech Videos