Sponsored Content
Top Forums Shell Programming and Scripting Rotating logs in Perl without message loss Post 302337449 by w1r3d on Friday 24th of July 2009 07:06:29 AM
Old 07-24-2009
Rotating logs in Perl without message loss

(I'm aware log rotation is a common subject, but I tried searching and couldn't find an answer)

For some time now, I've been using the Logfile::Rotate module to rotate logs in a log-monitoring script. So far, I haven't experienced any problems, and it works great because I can use it in Linux and Windows (mainly using Red Hat, and XP/2003). Well, it was brought to my attention by a co-worker that it is possible to lose log data using the "copy and truncate" method that the module uses. He mentioned that traditionally, in Linux, you do "move then send HUP signal to process". It seems like that is the preferred method (from what I see by searching online), but that would make it difficult for the script to port accross multiple OS's like it does now.

I guess my question is - what are your thoughts on this? I guess I thought this module worked perfectly, but this is the point he brings:

Code:
    ## copy current to next incremental
    $next = "${currn}.1";
    copy ($curr, $next);        

    ## preserve permissions and status
    if ( $self->{'Persist'} eq 'yes' ){
        my @stat = stat $curr;
        chmod( $stat[2], $next ) or carp "error: chmod failed: ($next)";
        utime( $stat[8], $stat[9], $next ) or carp "error: failed: ($next)";
        chown( $stat[4], $stat[5], $next ) or carp "error: chown failed: ($next)";
    }

    # now truncate the file
    if( $self->{'Flock'} eq 'yes' )
    {
        truncate $curr,0 or croak "error: could not truncate $curr: $!"; }
    else{
        local(*IN);
        open(IN, "+>$self->{'File'}") 
            or croak "error: could not truncate $curr: $!";
    }

The module does a "flock" on the file, which is an advisory lock on Linux. If the application that is writing to the current log file doesn't flock, then the flock on the rotation module is a no-op.

He mentioned that if the rotation module is task switched by the OS between the 'copy' on the third line and the 'truncate' or 'open' lines, log messages will be lost.

I am not very familiar with the way Linux works, and what I find online are mostly examples (code) of log rotation, and not actual explanations. Any insight on this subject will be greatly appreciated.

Thanks in advance!

edit: did some more testing, and was able to confirm (very minor) log loss with Linux's chatty auditd. I now have it so auditd rotates its own log files, and my script only monitors it. I'll have to come up with log file rotation schemes for all the different files I'm monitoring. Ugh. :-\

Last edited by w1r3d; 07-24-2009 at 08:56 AM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Message Logs Error

I'm about 5 months new on an 5 year old unix system. If anyone can help me identify what causing the below errors i'd really appreciate it! unix: WARNING: /pci@1f,0/pci@1,1/ide@3/dad@1,0 (dad1): Uncorrectable data Error: Block 57e10 Unix: WARNING: /pci@1f,0/pci@1,1/ide@3/dad@1,0 (dad1):... (1 Reply)
Discussion started by: ByasB
1 Replies

2. UNIX for Advanced & Expert Users

ppp errror message in logs

I am getting this message in the log file. Apr 29 15:32:02 router ppp: Warning: Label COPYRIGHT rejected -direct connection: Configuration label not found This repeats every so often, the link is up however...Any ideas why i am getting this. Its freebsd 6.1 and pppoE. Frank (1 Reply)
Discussion started by: frankkahle
1 Replies

3. Shell Programming and Scripting

Perl script to rotate logs

I have a shell script that will gzip/tar/archive application logs that are over 20 days old which works just fine, but I would like to convert to a Perl script. Problem is, I'm a beginner with Perl and all attempts so far have failed. Basicaly I have a log dir /app/logs that contains several... (18 Replies)
Discussion started by: theninja
18 Replies

4. UNIX for Dummies Questions & Answers

Perl Scripting for monitoring logs

Hi, I am new to perl. I want to write a perl script to monitor logs. Where i want to monitor exceptions logged or any kind of error strings. I have a dir(On Solaris) with multiple log file which keeps rolling to .gz file after some time in that same dir. These logs files size keeps on... (1 Reply)
Discussion started by: solitare123
1 Replies

5. Shell Programming and Scripting

Concatenate Logs - Perl Question

Hi All, I am fresh to perl and had been using shell scripting in my past experiences. In my part of perl program, i am trying to run a application command ccm stop, which should give some string output as the result. The output (error or sucess) has to be returned to an exisiting log file.... (4 Replies)
Discussion started by: ganga.dharan
4 Replies

6. Shell Programming and Scripting

script to grep outofmemory message in logs

I have prepare script to grep for outofmemory messages in the logs. I need help in modifying script. I have implemented small logic. The outofmemory messages form six logs will store in variables. var1=`grep -i outofmemory $tomcat1logs | sed -n '$p'| sed -n -e "s/.*\(outofmemory\).*/\1/p"`... (6 Replies)
Discussion started by: coolguyamy
6 Replies

7. Shell Programming and Scripting

Pattern count on rotating logs for the past 1 Hr

Hi All, I have a requirement to write a shell script to search the logs in past 1 hour and extract some pattern from it and count it cumulatively to a file. The problem which I'm facing here is - logs rotates on size basis, say if size of log reaches 5 MB then new log will be generated and... (7 Replies)
Discussion started by: Gem_In_I
7 Replies

8. Shell Programming and Scripting

Unable to get full message text from Windows Event Logs

Hi all, . I am developing a log monitoring solution in perl for Windows I am using the CPAN module Win32 ::EventLog (0.076) version for getting the events from windows. The problem which I am facing now is all the Windows 2008 machines are upgraded with Service pack2 from then I couldn’t able... (2 Replies)
Discussion started by: kar_333
2 Replies

9. Shell Programming and Scripting

If I ran perl script again,old logs should move with today date and new logs should generate.

Appreciate help for the below issue. Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt.............. 1)if logs exist it should move the logs with extention... (1 Reply)
Discussion started by: Sanjeev G
1 Replies
ROTATELOGS(8)							    rotatelogs							     ROTATELOGS(8)

NAME
rotatelogs - Piped logging program to rotate Apache logs SYNOPSIS
rotatelogs [ -l ] [ -f ] logfile rotationtime|filesizeM [ offset ] SUMMARY
rotatelogs is a simple program for use in conjunction with Apache's piped logfile feature. It supports rotation based on a time interval or maximum size of the log. OPTIONS
-l Causes the use of local time rather than GMT as the base for the interval or for strftime(3) formatting with size-based rotation. Note that using -l in an environment which changes the GMT offset (such as for BST or DST) can lead to unpredictable results! -f Causes the logfile to be opened immediately, as soon as rotatelogs starts, instead of waiting for the first logfile entry to be read (for non-busy sites, there may be a substantial delay between when the server is started and when the first request is handled, meaning that the associated logfile does not "exist" until then, which causes problems from some automated logging tools). Available in version 2.2.9 and later. logfile The path plus basename of the logfile. If logfile includes any '%' characters, it is treated as a format string for strftime(3). Otherwise, the suffix .nnnnnnnnnn is automatically added and is the time in seconds. Both formats compute the start time from the beginning of the current period. For example, if a rotation time of 86400 is specified, the hour, minute, and second fields created from the strftime(3) format will all be zero, referring to the beginning of the current 24-hour period (midnight). rotationtime The time between log file rotations in seconds. The rotation occurs at the beginning of this interval. For example, if the rotation time is 3600, the log file will be rotated at the beginning of every hour; if the rotation time is 86400, the log file will be rotated every night at midnight. (If no data is logged during an interval, no file will be created.) filesizeM The maximum file size in megabytes followed by the letter M to specify size rather than time. offset The number of minutes offset from UTC. If omitted, zero is assumed and UTC is used. For example, to use local time in the zone UTC -5 hours, specify a value of -300 for this argument. In most cases, -l should be used instead of specifying an offset. EXAMPLES
CustomLog "|bin/rotatelogs /var/logs/logfile 86400" common This creates the files /var/logs/logfile.nnnn where nnnn is the system time at which the log nominally starts (this time will always be a multiple of the rotation time, so you can synchronize cron scripts with it). At the end of each rotation time (here after 24 hours) a new log is started. CustomLog "|bin/rotatelogs -l /var/logs/logfile.%Y.%m.%d 86400" common This creates the files /var/logs/logfile.yyyy.mm.dd where yyyy is the year, mm is the month, and dd is the day of the month. Logging will switch to a new file every day at midnight, local time. CustomLog "|bin/rotatelogs /var/logs/logfile 5M" common This configuration will rotate the logfile whenever it reaches a size of 5 megabytes. ErrorLog "|bin/rotatelogs /var/logs/errorlog.%Y-%m-%d-%H_%M_%S 5M" This configuration will rotate the error logfile whenever it reaches a size of 5 megabytes, and the suffix to the logfile name will be cre- ated of the form errorlog.YYYY-mm-dd-HH_MM_SS. PORTABILITY
The following logfile format string substitutions should be supported by all strftime(3) implementations, see the strftime(3) man page for library-specific extensions. o %A - full weekday name (localized) o %a - 3-character weekday name (localized) o %B - full month name (localized) o %b - 3-character month name (localized) o %c - date and time (localized) o %d - 2-digit day of month o %H - 2-digit hour (24 hour clock) o %I - 2-digit hour (12 hour clock) o %j - 3-digit day of year o %M - 2-digit minute o %m - 2-digit month o %p - am/pm of 12 hour clock (localized) o %S - 2-digit second o %U - 2-digit week of year (Sunday first day of week) o %W - 2-digit week of year (Monday first day of week) o %w - 1-digit weekday (Sunday first day of week) o %X - time (localized) o %x - date (localized) o %Y - 4-digit year o %y - 2-digit year o %Z - time zone name o %% - literal `%' Apache HTTP Server 2008-05-10 ROTATELOGS(8)
All times are GMT -4. The time now is 08:16 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy