I have run into a problem where about a dozen machines, all the same x86_64 2.6.12 GNU/Linux. For some reason these machines will fill up their /var partition (10G), because their logs never get rotated... Unfortunately, there is no error message from logrotate (would be in /var/log/messages) and the last time logrotate ran (according to /var/log/logrotate.log) was August 11, 2008.
This unfortunately is somewhat of a puzzling problem, making it feel (however unlikely) that this is actually a problem with cron, and not log-rotate. IE. cron failed first, which made logrotate never happen. I had originally thought that log-rotate just couldn't do its job because there was no space available for the rotation to occur. But even when I gave it enough space, cron just never ran the logrotate script. On top of that, I even added my own "append timestamp to file" script to the cron.hourly, and it never got run either. As a test, on one of the servers, I restarted cron... BINGO, it fixed the whole thing.
I could care less about the fact that I can fix it. I want to prevent this from ever happening in the field. In my lab is one thing, but in the field, there goes all my logs for debugging.
Here is a sanity check I ran (just proving that cron was functioning and stopped functioning - all within the same up-time):
Any help?
Can you be sure that the jobs aren't being run if you are relying on logs stored in /var to check? If /var is 100% full, it's quite possible that the jobs *are* running, but simply can't add a log entry to say so. Did your "append timestamp to file" test script use a file on another filesystem?
Having said that, I think it's quite likely that cron would stop working once /var becomes 100% full... there must be some pretty big logs being generated somewhere to fill a 9.5GB /var?
The "append timestamp to file" test used /tmp... which is on /dev/sda2.
If you look at the output from "ll /var/log/cron*" you will see that messages continued to be sent to /var/log/cron from after August 10. The last line of the cron.1 file confirms that August 10 was when the last logrotate happened. Given that cron logs should be relatively consistent, one would expect the next cron log rotation to occur on August 17... but it did not. Also, because of the size (~14k) of the cron log, there is a guarantee that the logrotate did not fail (on the 17th) due to the filesystem being full... instead logrotate just never ran.
According to /var/log/logrotate.log, the last time that logrotate ran was:
1218492062 which translates to : August 11th, 2008 10:01:02 PM
As for the logs being generated, yeah, they are big, but its normal for my app with debug on max...
Have you established whether cron stops running all jobs when it's in this state? Just the cron.hourly? Or just jobs for a specific user (i.e. root in this case)? Also, are there any other cron jobs stuck at the time?
Have you tried strace-ing a crond that's stuck like this? Also, does a pkill -HUP crond "wake" it up again?
It might also be worth running cron with some debugging options, although they don't seem to be very well documented. The best reference (for Vixie cron) I've found so far is:
I tried to use GDB but it would appear that the symbols are stripped from the crond binary, as the backtrace only provided the following:
I HUPed the cron daemon, and waited to see if the test script ran, but unfortunately no... even though it is theoretically writing to someplace other than the /var partition (aka. /tmp)
As for strace, I was unaware that I could attach strace to a process that is already running... if that is so, can you reference an example? I will gladly kick it off as a test.
Hi, I have a problem with logrotate at Centos 7.
My logrotate is configured with "rotate 0" to Apache logs, so it should never keep logs when rotating, just removing them and replacing by new empty ones at every rotation. But for some reason, once in a while, I see that logrotate is creating... (0 Replies)
Hi Admins.
I have installed logrotate rpm on Aix 6.1.
After the installation of rpm, I don't find /etc/logrotate.conf file and /etc/logrotate.d dir .
The config file is located in /opt/freeware/etc/logrotate.conf.
When I ran
logrotate -v /opt/freeware/etc/logrotate.conf
I get below... (2 Replies)
I have written script which is working in Home directory perfectly and also compressing log files and rotating correctly. But, when i try to run script for /var/log/ i am able to get compressed log files but not able to get rotation of compressed log files. Please suggest.
I am using below command... (5 Replies)
Hi guys,
I've got two separate logrotates I'd like to run, one for Tomcat and one for Apache, but I'd like to run the Tomcat one daily and the Apache one weekly. Now, the logrotate itself is working fine, but although I have 'daily' in Tomcat, and 'weekly' in the Apache one, the latter is... (2 Replies)
Hi,
I have the following configuration file:
/logs/system/mindundi/* {
rotate 0
daily
missingok
sharedscripts
postrotate
find /logs/system/mindundi/ -name "*log" -mtime +15 -exec /bin/rm -f {} \;
endscript
}
I want to save only... (6 Replies)
Hi there,
I want to rotate the logfiles which are located in /var/log/jboss/tomcat*
so I have created a file named as 'tomat' in /etc/logrotate.d/tomcat with the following content.
# cat /etc/logrotate.d/tomcat
/var/log/jboss/tomcat_access_log*.log {
daily
nocreate
... (2 Replies)
Hi all,
I have configured logrotate to logorotate every 12 hour. The configurations are as follows.
/etc/cron.d/config
-------------------------
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=""
HOME=/root
0 */12 * * * root logrotate /etc/logrotate.d/test
... (1 Reply)