Sponsored Content
Top Forums UNIX for Advanced & Expert Users cron/logrotate chicken and egg problem Post 302235278 by jjinno on Thursday 11th of September 2008 02:13:22 PM
Old 09-11-2008
cron/logrotate chicken and egg problem

I have run into a problem where about a dozen machines, all the same x86_64 2.6.12 GNU/Linux. For some reason these machines will fill up their /var partition (10G), because their logs never get rotated... Unfortunately, there is no error message from logrotate (would be in /var/log/messages) and the last time logrotate ran (according to /var/log/logrotate.log) was August 11, 2008.

This unfortunately is somewhat of a puzzling problem, making it feel (however unlikely) that this is actually a problem with cron, and not log-rotate. IE. cron failed first, which made logrotate never happen. I had originally thought that log-rotate just couldn't do its job because there was no space available for the rotation to occur. But even when I gave it enough space, cron just never ran the logrotate script. On top of that, I even added my own "append timestamp to file" script to the cron.hourly, and it never got run either. As a test, on one of the servers, I restarted cron... BINGO, it fixed the whole thing.

I could care less about the fact that I can fix it. I want to prevent this from ever happening in the field. In my lab is one thing, but in the field, there goes all my logs for debugging.

Here is a sanity check I ran (just proving that cron was functioning and stopped functioning - all within the same up-time):
Code:
[root@babylon5 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             9.5G  3.7G  5.3G  42% /
/dev/sda1             190M   18M  163M  10% /boot
/dev/shm              2.0G  4.0K  2.0G   1% /dev/shm
/dev/sda6              44G  8.4G   33G  21% /stuff
/dev/sda5             9.5G  9.5G     0 100% /var

[root@babylon5 ~]# ls -lut /etc/init.d/crond
-rwxr-xr-x  1 root root 1904 Aug 11 15:14 /etc/init.d/crond

[root@babylon5 ~]# ls -lut /etc/crontab
-rw-r--r--  1 root root 255 Jul 15 10:19 /etc/crontab

[root@babylon5 ~]# grep "Aug 11 15" /var/log/cron
Aug 11 15:01:02 babylon5 crond[9429]: (root) CMD (run-parts /etc/cron.hourly)

[root@babylon5 ~]# ps auxwww|grep cr[o]n
root      3014  0.0  0.0  60404  1296 ?        Ss   Jul15   0:00 crond

[root@babylon5 ~]# uptime
 13:33:34 up 57 days,  3:15,  1 user,  load average: 15.63, 15.37, 15.29

[root@babylon5 ~]# grep "STARTUP" /var/log/cron*
/var/log/cron.4:Jul 14 14:16:44 babylon5 crond[3156]: (CRON) STARTUP (V5.0)
/var/log/cron.4:Jul 15 08:53:45 babylon5 crond[3008]: (CRON) STARTUP (V5.0)
/var/log/cron.4:Jul 15 10:19:35 babylon5 crond[3014]: (CRON) STARTUP (V5.0)

[root@babylon5 ~]# tail -n 1 /var/log/cron.4
Jul 20 00:01:01 babylon5 crond[15555]: (root) CMD (run-parts /etc/cron.hourly)

[root@babylon5 ~]# tail -n 1 /var/log/cron.3
Jul 27 01:01:02 babylon5 crond[27941]: (root) CMD (run-parts /etc/cron.hourly)

[root@babylon5 ~]# tail -n 1 /var/log/cron.2
Aug  3 01:01:01 babylon5 crond[17755]: (root) CMD (run-parts /etc/cron.hourly)

[root@babylon5 ~]# tail -n 1 /var/log/cron.1
Aug 10 01:01:02 babylon5 crond[29697]: (root) CMD (run-parts /etc/cron.hourly)

[root@babylon5 ~]# tail -n 1 /var/log/cron
Sep 10 13:01:01 babylon5 crond[22186]: (root) MAIL (mailed 146 bytes of output but got status 0x007f )

[root@babylon5 ~]# ll /var/log/cron*
-rw-r--r--  1 root root 138502 Sep 10 13:01 /var/log/cron
-rw-------  1 root root  14407 Aug 10 01:01 /var/log/cron.1
-rw-------  1 root root  14581 Aug  3 01:01 /var/log/cron.2
-rw-------  1 root root  14483 Jul 27 01:01 /var/log/cron.3
-rw-------  1 root root  14513 Jul 20 00:01 /var/log/cron.4

Any help?
 

8 More Discussions You Might Find Interesting

1. Red Hat

Problem seen with logrotate

Hi all, I have configured logrotate to logorotate every 12 hour. The configurations are as follows. /etc/cron.d/config ------------------------- SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO="" HOME=/root 0 */12 * * * root logrotate /etc/logrotate.d/test ... (1 Reply)
Discussion started by: rsravi74
1 Replies

2. UNIX for Advanced & Expert Users

logrotate with /etc/logrotate.conf file

Hi there, I want to rotate the logfiles which are located in /var/log/jboss/tomcat* so I have created a file named as 'tomat' in /etc/logrotate.d/tomcat with the following content. # cat /etc/logrotate.d/tomcat /var/log/jboss/tomcat_access_log*.log { daily nocreate ... (2 Replies)
Discussion started by: skmdu
2 Replies

3. UNIX for Advanced & Expert Users

Logrotate configuration problem

Hi, I have the following configuration file: /logs/system/mindundi/* { rotate 0 daily missingok sharedscripts postrotate find /logs/system/mindundi/ -name "*log" -mtime +15 -exec /bin/rm -f {} \; endscript } I want to save only... (6 Replies)
Discussion started by: mitchbcn
6 Replies

4. UNIX for Dummies Questions & Answers

logrotate and cron.daily/weekly

Hi guys, I've got two separate logrotates I'd like to run, one for Tomcat and one for Apache, but I'd like to run the Tomcat one daily and the Apache one weekly. Now, the logrotate itself is working fine, but although I have 'daily' in Tomcat, and 'weekly' in the Apache one, the latter is... (2 Replies)
Discussion started by: jimbob01
2 Replies

5. Shell Programming and Scripting

Logrotate - I am not able to rotate files using logrotate

I have written script which is working in Home directory perfectly and also compressing log files and rotating correctly. But, when i try to run script for /var/log/ i am able to get compressed log files but not able to get rotation of compressed log files. Please suggest. I am using below command... (5 Replies)
Discussion started by: VSom007
5 Replies

6. AIX

Logrotate - /etc/logrotate.conf does't exist

Hi Admins. I have installed logrotate rpm on Aix 6.1. After the installation of rpm, I don't find /etc/logrotate.conf file and /etc/logrotate.d dir . The config file is located in /opt/freeware/etc/logrotate.conf. When I ran logrotate -v /opt/freeware/etc/logrotate.conf I get below... (2 Replies)
Discussion started by: snchaudhari2
2 Replies

7. Shell Programming and Scripting

Problem with logrotate

Hi, I have a problem with logrotate at Centos 7. My logrotate is configured with "rotate 0" to Apache logs, so it should never keep logs when rotating, just removing them and replacing by new empty ones at every rotation. But for some reason, once in a while, I see that logrotate is creating... (0 Replies)
Discussion started by: dado000
0 Replies

8. Shell Programming and Scripting

How to disable cron emails, but only for logrotate only not for other logs?

Guys, is there a script or command? how to disable cron emails, but only for logrotate only not for other logs (3 Replies)
Discussion started by: kenshinhimura
3 Replies
CRON(8) 						    BSD System Manager's Manual 						   CRON(8)

NAME
cron -- daemon to execute scheduled commands (ISC Cron V4.1) SYNOPSIS
cron [-n] [-x debugflags] DESCRIPTION
cron is normally started during system boot by rc.d(8) framework, if cron is switched on in rc.conf(5). It will return immediately so you don't have to start it with '&'. cron searches /var/cron/tabs for crontab files which are named after accounts in /etc/passwd. Crontabs found are loaded into memory. cron also searches for /etc/crontab which is in a different format (see crontab(5)). Finally cron looks for crontabs in /etc/cron.d if it exists, and executes each file as a crontab. When cron looks in a directory for crontabs (either in /var/cron/tabs or /etc/cron.d) it will not process files that: - Start with a '.' or a '#'. - End with a '~' or with ``.rpmsave'', ``.rpmorig'', or ``.rpmnew''. - Are of zero length. - Their length is greater than MAXNAMLEN. cron then wakes up every minute, examining all stored crontabs, checking each command to see if it should be run in the current minute. When executing commands, any output is mailed to the owner of the crontab (or to the user named in the MAILTO environment variable in the crontab, if such exists). Events such as START and FINISH are recorded in the /var/log/cron log file with date and time details. This information is useful for a num- ber of reasons, such as determining the amount of time required to run a particular job. By default, root has an hourly job that rotates these log files with compression to preserve disk space. Additionally, cron checks each minute to see if its spool directory's modtime (or the modtime on /etc/crontab or /etc/cron.d) has changed, and if it has, cron will then examine the modtime on all crontabs and reload those which have changed. Thus cron need not be restarted when- ever a crontab file is modified. Note that the crontab(1) command updates the modtime of the spool directory whenever it changes a crontab. The following options are available: -x This flag turns on some debugging flags. debugflags is comma-separated list of debugging flags to turn on. If a flag is turned on, cron writes some additional debugging information to system log during its work. Available debugging flags are: sch scheduling proc process control pars parsing load database loading misc miscellaneous test test mode - do not actually execute any commands bit show how various bits are set (long) ext print extended debugging information -n Stay in the foreground and don't daemonize cron. Daylight Saving Time and other time changes Local time changes of less than three hours, such as those caused by the start or end of Daylight Saving Time, are handled specially. This only applies to jobs that run at a specific time and jobs that are run with a granularity greater than one hour. Jobs that run more fre- quently are scheduled normally. If time has moved forward, those jobs that would have run in the interval that has been skipped will be run immediately. Conversely, if time has moved backward, care is taken to avoid running jobs twice. Time changes of more than 3 hours are considered to be corrections to the clock or timezone, and the new time is used immediately. SIGNALS
On receipt of a SIGHUP, the cron daemon will close and reopen its log file. This is useful in scripts which rotate and age log files. Natu- rally this is not relevant if cron was built to use syslog(3). FILES
/var/cron/tabs cron spool directory /etc/crontab system crontab file /etc/cron.d/ system crontab directory /var/log/cron log file for cron events SEE ALSO
crontab(1), crontab(5) AUTHORS
Paul Vixie <vixie@isc.org> BSD
October 12, 2011 BSD
All times are GMT -4. The time now is 08:01 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy