Sponsored Content
Top Forums Shell Programming and Scripting Rotating logs in Perl without message loss Post 302337449 by w1r3d on Friday 24th of July 2009 07:06:29 AM
Old 07-24-2009
Rotating logs in Perl without message loss

(I'm aware log rotation is a common subject, but I tried searching and couldn't find an answer)

For some time now, I've been using the Logfile::Rotate module to rotate logs in a log-monitoring script. So far, I haven't experienced any problems, and it works great because I can use it in Linux and Windows (mainly using Red Hat, and XP/2003). Well, it was brought to my attention by a co-worker that it is possible to lose log data using the "copy and truncate" method that the module uses. He mentioned that traditionally, in Linux, you do "move then send HUP signal to process". It seems like that is the preferred method (from what I see by searching online), but that would make it difficult for the script to port accross multiple OS's like it does now.

I guess my question is - what are your thoughts on this? I guess I thought this module worked perfectly, but this is the point he brings:

Code:
    ## copy current to next incremental
    $next = "${currn}.1";
    copy ($curr, $next);        

    ## preserve permissions and status
    if ( $self->{'Persist'} eq 'yes' ){
        my @stat = stat $curr;
        chmod( $stat[2], $next ) or carp "error: chmod failed: ($next)";
        utime( $stat[8], $stat[9], $next ) or carp "error: failed: ($next)";
        chown( $stat[4], $stat[5], $next ) or carp "error: chown failed: ($next)";
    }

    # now truncate the file
    if( $self->{'Flock'} eq 'yes' )
    {
        truncate $curr,0 or croak "error: could not truncate $curr: $!"; }
    else{
        local(*IN);
        open(IN, "+>$self->{'File'}") 
            or croak "error: could not truncate $curr: $!";
    }

The module does a "flock" on the file, which is an advisory lock on Linux. If the application that is writing to the current log file doesn't flock, then the flock on the rotation module is a no-op.

He mentioned that if the rotation module is task switched by the OS between the 'copy' on the third line and the 'truncate' or 'open' lines, log messages will be lost.

I am not very familiar with the way Linux works, and what I find online are mostly examples (code) of log rotation, and not actual explanations. Any insight on this subject will be greatly appreciated.

Thanks in advance!

edit: did some more testing, and was able to confirm (very minor) log loss with Linux's chatty auditd. I now have it so auditd rotates its own log files, and my script only monitors it. I'll have to come up with log file rotation schemes for all the different files I'm monitoring. Ugh. :-\

Last edited by w1r3d; 07-24-2009 at 08:56 AM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Message Logs Error

I'm about 5 months new on an 5 year old unix system. If anyone can help me identify what causing the below errors i'd really appreciate it! unix: WARNING: /pci@1f,0/pci@1,1/ide@3/dad@1,0 (dad1): Uncorrectable data Error: Block 57e10 Unix: WARNING: /pci@1f,0/pci@1,1/ide@3/dad@1,0 (dad1):... (1 Reply)
Discussion started by: ByasB
1 Replies

2. UNIX for Advanced & Expert Users

ppp errror message in logs

I am getting this message in the log file. Apr 29 15:32:02 router ppp: Warning: Label COPYRIGHT rejected -direct connection: Configuration label not found This repeats every so often, the link is up however...Any ideas why i am getting this. Its freebsd 6.1 and pppoE. Frank (1 Reply)
Discussion started by: frankkahle
1 Replies

3. Shell Programming and Scripting

Perl script to rotate logs

I have a shell script that will gzip/tar/archive application logs that are over 20 days old which works just fine, but I would like to convert to a Perl script. Problem is, I'm a beginner with Perl and all attempts so far have failed. Basicaly I have a log dir /app/logs that contains several... (18 Replies)
Discussion started by: theninja
18 Replies

4. UNIX for Dummies Questions & Answers

Perl Scripting for monitoring logs

Hi, I am new to perl. I want to write a perl script to monitor logs. Where i want to monitor exceptions logged or any kind of error strings. I have a dir(On Solaris) with multiple log file which keeps rolling to .gz file after some time in that same dir. These logs files size keeps on... (1 Reply)
Discussion started by: solitare123
1 Replies

5. Shell Programming and Scripting

Concatenate Logs - Perl Question

Hi All, I am fresh to perl and had been using shell scripting in my past experiences. In my part of perl program, i am trying to run a application command ccm stop, which should give some string output as the result. The output (error or sucess) has to be returned to an exisiting log file.... (4 Replies)
Discussion started by: ganga.dharan
4 Replies

6. Shell Programming and Scripting

script to grep outofmemory message in logs

I have prepare script to grep for outofmemory messages in the logs. I need help in modifying script. I have implemented small logic. The outofmemory messages form six logs will store in variables. var1=`grep -i outofmemory $tomcat1logs | sed -n '$p'| sed -n -e "s/.*\(outofmemory\).*/\1/p"`... (6 Replies)
Discussion started by: coolguyamy
6 Replies

7. Shell Programming and Scripting

Pattern count on rotating logs for the past 1 Hr

Hi All, I have a requirement to write a shell script to search the logs in past 1 hour and extract some pattern from it and count it cumulatively to a file. The problem which I'm facing here is - logs rotates on size basis, say if size of log reaches 5 MB then new log will be generated and... (7 Replies)
Discussion started by: Gem_In_I
7 Replies

8. Shell Programming and Scripting

Unable to get full message text from Windows Event Logs

Hi all, . I am developing a log monitoring solution in perl for Windows I am using the CPAN module Win32 ::EventLog (0.076) version for getting the events from windows. The problem which I am facing now is all the Windows 2008 machines are upgraded with Service pack2 from then I couldn’t able... (2 Replies)
Discussion started by: kar_333
2 Replies

9. Shell Programming and Scripting

If I ran perl script again,old logs should move with today date and new logs should generate.

Appreciate help for the below issue. Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt.............. 1)if logs exist it should move the logs with extention... (1 Reply)
Discussion started by: Sanjeev G
1 Replies
NEWSYSLOG(8)						    BSD System Manager's Manual 					      NEWSYSLOG(8)

NAME
newsyslog -- maintain system log files to manageable sizes SYNOPSIS
newsyslog [-CFNnrsv] [-R tagname] [-a directory] [-d directory] [-f config_file] [file ...] DESCRIPTION
The newsyslog utility should be scheduled to run periodically by cron(8). When it is executed it archives log files if necessary. If a log file is determined to require archiving, newsyslog rearranges the files so that ``logfile'' is empty, ``logfile.0'' has the last period's logs in it, ``logfile.1'' has the next to last period's logs in it, and so on, up to a user-specified number of archived logs. Optionally the archived logs can be compressed to save space. A log can be archived for three reasons: 1. It is larger than the configured size (in kilobytes). 2. A configured number of hours have elapsed since the log was last archived. 3. This is the specific configured hour for rotation of the log. The granularity of newsyslog is dependent on how often it is scheduled to run by cron(8). Since the program is quite fast, it may be sched- uled to run every hour without any ill effects, and mode three (above) assumes that this is so. OPTIONS
The following options can be used with newsyslog: -f config_file Instruct newsyslog to use config_file instead of /etc/newsyslog.conf and /etc/newsyslog.d/*.conf for its configuration file. -a directory Specify a directory into which archived log files will be written. If a relative path is given, it is appended to the path of each log file and the resulting path is used as the directory into which the archived log for that log file will be written. If an abso- lute path is given, all archived logs are written into the given directory. If any component of the path directory does not exist, it will be created when newsyslog is run. -d directory Specify a directory which all log files will be relative to. To allow archiving of logs outside the root, the directory passed to the -a option is unaffected. -v Place newsyslog in verbose mode. In this mode it will print out each log and its reasons for either trimming that log or skipping it. -n Cause newsyslog not to trim the logs, but to print out what it would do if this option were not specified. -r Remove the restriction that newsyslog must be running as root. Of course, newsyslog will not be able to send a HUP signal to syslogd(8) so this option should only be used in debugging. -s Specify that newsyslog should not send any signals to any daemon processes that it would normally signal when rotating a log file. For any log file which is rotated, this option will usually also mean the rotated log file will not be compressed if there is a dae- mon which would have been signalled without this option. However, this option is most likely to be useful when specified with the -R option, and in that case the compression will be done. -C If specified once, then newsyslog will create any log files which do not exist, and which have the C flag specified in their config file entry. If specified multiple times, then newsyslog will create all log files which do not already exist. If log files are given on the command-line, then the -C or -CC will only apply to those specific log files. -F Force newsyslog to trim the logs, even if the trim conditions have not been met. This option is useful for diagnosing system prob- lems by providing you with fresh logs that contain only the problems. -N Do not perform any rotations. This option is intended to be used with the -C or -CC options when creating log files is the only objective. -R tagname Specify that newsyslog should rotate a given list of files, even if trim conditions are not met for those files. The tagname is only used in the messages written to the log files which are rotated. This differs from the -F option in that one or more log files must also be specified, so that newsyslog will only operate on those specific files. This option is mainly intended for the daemons or programs which write some log files, and want to trigger a rotate based on their own criteria. With this option they can execute newsyslog to trigger the rotate when they want it to happen, and still give the system administrator a way to specify the rules of rotation (such as how many backup copies are kept, and what kind of compression is done). When a daemon does execute newsyslog with the -R option, it should make sure all of the log files are closed before calling newsyslog, and then it should re-open the files after newsyslog returns. Usually the calling process will also want to specify the -s option, so newsyslog will not send a signal to the very process which called it to force the rotate. Skipping the signal step will also mean that newsyslog will return faster, since newsyslog normally waits a few seconds after any signal that is sent. If additional command line arguments are given, newsyslog will only examine log files that match those arguments; otherwise, it will examine all files listed in the configuration file(s). FILES
/etc/newsyslog.conf newsyslog configuration file /etc/newsyslog.d/ newsyslog configuration directory COMPATIBILITY
Previous versions of the newsyslog utility used the dot (``.'') character to distinguish the group name. Beginning with FreeBSD 3.3, this has been changed to a colon (``:'') character so that user and group names may contain the dot character. The dot (``.'') character is still accepted for backwards compatibility. HISTORY
The newsyslog utility originated from NetBSD and first appeared in FreeBSD 2.2. AUTHORS
Theodore Ts'o, MIT Project Athena Copyright 1987, Massachusetts Institute of Technology SEE ALSO
bzip2(1), gzip(1), syslog(3), newsyslog.conf(5), chown(8), syslogd(8) BUGS
Does not yet automatically read the logs to find security breaches. BSD
February 24, 2005 BSD
All times are GMT -4. The time now is 07:19 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy