(I'm aware log rotation is a common subject, but I tried searching and couldn't find an answer)
For some time now, I've been using the Logfile::Rotate module to rotate logs in a log-monitoring script. So far, I haven't experienced any problems, and it works great because I can use it in Linux and Windows (mainly using Red Hat, and XP/2003). Well, it was brought to my attention by a co-worker that it is possible to lose log data using the "copy and truncate" method that the module uses. He mentioned that traditionally, in Linux, you do "move then send HUP signal to process". It seems like that is the preferred method (from what I see by searching online), but that would make it difficult for the script to port accross multiple OS's like it does now.
I guess my question is - what are your thoughts on this? I guess I thought this module worked perfectly, but this is the point he brings:
The module does a "flock" on the file, which is an advisory lock on Linux. If the application that is writing to the current log file doesn't flock, then the flock on the rotation module is a no-op.
He mentioned that if the rotation module is task switched by the OS between the 'copy' on the third line and the 'truncate' or 'open' lines, log messages will be lost.
I am not very familiar with the way Linux works, and what I find online are mostly examples (code) of log rotation, and not actual explanations. Any insight on this subject will be greatly appreciated.
Thanks in advance!
edit: did some more testing, and was able to confirm (very minor) log loss with Linux's chatty auditd. I now have it so auditd rotates its own log files, and my script only monitors it. I'll have to come up with log file rotation schemes for all the different files I'm monitoring. Ugh. :-\
I'm about 5 months new on an 5 year old unix system. If anyone can help me identify what causing the below errors i'd really appreciate it!
unix: WARNING: /pci@1f,0/pci@1,1/ide@3/dad@1,0 (dad1):
Uncorrectable data Error: Block 57e10
Unix: WARNING: /pci@1f,0/pci@1,1/ide@3/dad@1,0 (dad1):... (1 Reply)
I am getting this message in the log file.
Apr 29 15:32:02 router ppp: Warning: Label COPYRIGHT rejected -direct connection: Configuration label not found
This repeats every so often, the link is up however...Any ideas why i am getting this. Its freebsd 6.1 and pppoE.
Frank (1 Reply)
I have a shell script that will gzip/tar/archive application logs that are over 20 days old which works just fine, but I would like to convert to a Perl script. Problem is, I'm a beginner with Perl and all attempts so far have failed.
Basicaly I have a log dir /app/logs that contains several... (18 Replies)
Hi,
I am new to perl. I want to write a perl script to monitor logs. Where i want to monitor exceptions logged or any kind of error strings. I have a dir(On Solaris) with multiple log file which keeps rolling to .gz file after some time in that same dir. These logs files size keeps on... (1 Reply)
Hi All,
I am fresh to perl and had been using shell scripting in my past experiences.
In my part of perl program, i am trying to run a application command ccm stop, which should give some string output as the result. The output (error or sucess) has to be returned to an exisiting log file.... (4 Replies)
I have prepare script to grep for outofmemory messages in the logs. I need help in modifying script. I have implemented small logic. The outofmemory messages form six logs will store in variables.
var1=`grep -i outofmemory $tomcat1logs | sed -n '$p'| sed -n -e "s/.*\(outofmemory\).*/\1/p"`... (6 Replies)
Hi All,
I have a requirement to write a shell script to search the logs in past 1 hour and extract some pattern from it and count it cumulatively to a file.
The problem which I'm facing here is - logs rotates on size basis, say if size of log reaches 5 MB then new log will be generated and... (7 Replies)
Hi all,
. I am developing a log monitoring solution in perl for Windows I am using the CPAN module Win32 ::EventLog (0.076) version for getting the events from windows. The problem which I am facing now is all the Windows 2008 machines are upgraded with Service pack2 from then I couldn’t able... (2 Replies)
Appreciate help for the below issue.
Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt..............
1)if logs exist it should move the logs with extention... (1 Reply)
Discussion started by: Sanjeev G
1 Replies
LEARN ABOUT OSX
newsyslog
NEWSYSLOG(8) BSD System Manager's Manual NEWSYSLOG(8)NAME
newsyslog -- maintain system log files to manageable sizes
SYNOPSIS
newsyslog [-CFNnrsv] [-R tagname] [-a directory] [-d directory] [-f config_file] [file ...]
DESCRIPTION
The newsyslog utility should be scheduled to run periodically by cron(8). When it is executed it archives log files if necessary. If a log
file is determined to require archiving, newsyslog rearranges the files so that ``logfile'' is empty, ``logfile.0'' has the last period's
logs in it, ``logfile.1'' has the next to last period's logs in it, and so on, up to a user-specified number of archived logs. Optionally
the archived logs can be compressed to save space.
A log can be archived for three reasons:
1. It is larger than the configured size (in kilobytes).
2. A configured number of hours have elapsed since the log was last archived.
3. This is the specific configured hour for rotation of the log.
The granularity of newsyslog is dependent on how often it is scheduled to run by cron(8). Since the program is quite fast, it may be sched-
uled to run every hour without any ill effects, and mode three (above) assumes that this is so.
OPTIONS
The following options can be used with newsyslog:
-f config_file
Instruct newsyslog to use config_file instead of /etc/newsyslog.conf and /etc/newsyslog.d/*.conf for its configuration file.
-a directory
Specify a directory into which archived log files will be written. If a relative path is given, it is appended to the path of each
log file and the resulting path is used as the directory into which the archived log for that log file will be written. If an abso-
lute path is given, all archived logs are written into the given directory. If any component of the path directory does not exist,
it will be created when newsyslog is run.
-d directory
Specify a directory which all log files will be relative to. To allow archiving of logs outside the root, the directory passed to
the -a option is unaffected.
-v Place newsyslog in verbose mode. In this mode it will print out each log and its reasons for either trimming that log or skipping
it.
-n Cause newsyslog not to trim the logs, but to print out what it would do if this option were not specified.
-r Remove the restriction that newsyslog must be running as root. Of course, newsyslog will not be able to send a HUP signal to
syslogd(8) so this option should only be used in debugging.
-s Specify that newsyslog should not send any signals to any daemon processes that it would normally signal when rotating a log file.
For any log file which is rotated, this option will usually also mean the rotated log file will not be compressed if there is a dae-
mon which would have been signalled without this option. However, this option is most likely to be useful when specified with the -R
option, and in that case the compression will be done.
-C If specified once, then newsyslog will create any log files which do not exist, and which have the C flag specified in their config
file entry. If specified multiple times, then newsyslog will create all log files which do not already exist. If log files are
given on the command-line, then the -C or -CC will only apply to those specific log files.
-F Force newsyslog to trim the logs, even if the trim conditions have not been met. This option is useful for diagnosing system prob-
lems by providing you with fresh logs that contain only the problems.
-N Do not perform any rotations. This option is intended to be used with the -C or -CC options when creating log files is the only
objective.
-R tagname
Specify that newsyslog should rotate a given list of files, even if trim conditions are not met for those files. The tagname is only
used in the messages written to the log files which are rotated. This differs from the -F option in that one or more log files must
also be specified, so that newsyslog will only operate on those specific files. This option is mainly intended for the daemons or
programs which write some log files, and want to trigger a rotate based on their own criteria. With this option they can execute
newsyslog to trigger the rotate when they want it to happen, and still give the system administrator a way to specify the rules of
rotation (such as how many backup copies are kept, and what kind of compression is done). When a daemon does execute newsyslog with
the -R option, it should make sure all of the log files are closed before calling newsyslog, and then it should re-open the files
after newsyslog returns. Usually the calling process will also want to specify the -s option, so newsyslog will not send a signal to
the very process which called it to force the rotate. Skipping the signal step will also mean that newsyslog will return faster,
since newsyslog normally waits a few seconds after any signal that is sent.
If additional command line arguments are given, newsyslog will only examine log files that match those arguments; otherwise, it will examine
all files listed in the configuration file(s).
FILES
/etc/newsyslog.conf newsyslog configuration file
/etc/newsyslog.d/ newsyslog configuration directory
COMPATIBILITY
Previous versions of the newsyslog utility used the dot (``.'') character to distinguish the group name. Beginning with FreeBSD 3.3, this
has been changed to a colon (``:'') character so that user and group names may contain the dot character. The dot (``.'') character is still
accepted for backwards compatibility.
HISTORY
The newsyslog utility originated from NetBSD and first appeared in FreeBSD 2.2.
AUTHORS
Theodore Ts'o, MIT Project Athena
Copyright 1987, Massachusetts Institute of Technology
SEE ALSO bzip2(1), gzip(1), syslog(3), newsyslog.conf(5), chown(8), syslogd(8)BUGS
Does not yet automatically read the logs to find security breaches.
BSD February 24, 2005 BSD