Rotating logs in Perl without message loss

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Rotating logs in Perl without message loss
# 1  
Old 07-24-2009
Rotating logs in Perl without message loss

(I'm aware log rotation is a common subject, but I tried searching and couldn't find an answer)

For some time now, I've been using the Logfile::Rotate module to rotate logs in a log-monitoring script. So far, I haven't experienced any problems, and it works great because I can use it in Linux and Windows (mainly using Red Hat, and XP/2003). Well, it was brought to my attention by a co-worker that it is possible to lose log data using the "copy and truncate" method that the module uses. He mentioned that traditionally, in Linux, you do "move then send HUP signal to process". It seems like that is the preferred method (from what I see by searching online), but that would make it difficult for the script to port accross multiple OS's like it does now.

I guess my question is - what are your thoughts on this? I guess I thought this module worked perfectly, but this is the point he brings:

    ## copy current to next incremental
    $next = "${currn}.1";
    copy ($curr, $next);        

    ## preserve permissions and status
    if ( $self->{'Persist'} eq 'yes' ){
        my @stat = stat $curr;
        chmod( $stat[2], $next ) or carp "error: chmod failed: ($next)";
        utime( $stat[8], $stat[9], $next ) or carp "error: failed: ($next)";
        chown( $stat[4], $stat[5], $next ) or carp "error: chown failed: ($next)";

    # now truncate the file
    if( $self->{'Flock'} eq 'yes' )
        truncate $curr,0 or croak "error: could not truncate $curr: $!"; }
        open(IN, "+>$self->{'File'}") 
            or croak "error: could not truncate $curr: $!";

The module does a "flock" on the file, which is an advisory lock on Linux. If the application that is writing to the current log file doesn't flock, then the flock on the rotation module is a no-op.

He mentioned that if the rotation module is task switched by the OS between the 'copy' on the third line and the 'truncate' or 'open' lines, log messages will be lost.

I am not very familiar with the way Linux works, and what I find online are mostly examples (code) of log rotation, and not actual explanations. Any insight on this subject will be greatly appreciated.

Thanks in advance!

edit: did some more testing, and was able to confirm (very minor) log loss with Linux's chatty auditd. I now have it so auditd rotates its own log files, and my script only monitors it. I'll have to come up with log file rotation schemes for all the different files I'm monitoring. Ugh. :-\

Last edited by w1r3d; 07-24-2009 at 08:56 AM..
# 2  
Old 07-25-2009
That perl module does seem to be badly written. And most OS's already have a log rotation technique. Linux comes with a logrotate command. link: logrotate - Linux Command - Unix Command I wonder why you don't just use that. In the XP world it seems that the typical solution is to let software that keeps logs rotate its own logs. IIS takes this approach. It is not as flexible as logrotate, but I would investigate that before implementing a new solution.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

If I ran perl script again,old logs should move with today date and new logs should generate.

Appreciate help for the below issue. Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt.............. 1)if logs exist it should move the logs with extention... (1 Reply)
Discussion started by: Sanjeev G
1 Replies

2. Shell Programming and Scripting

Unable to get full message text from Windows Event Logs

Hi all, . I am developing a log monitoring solution in perl for Windows I am using the CPAN module Win32 ::EventLog (0.076) version for getting the events from windows. The problem which I am facing now is all the Windows 2008 machines are upgraded with Service pack2 from then I couldn’t able... (2 Replies)
Discussion started by: kar_333
2 Replies

3. Shell Programming and Scripting

Pattern count on rotating logs for the past 1 Hr

Hi All, I have a requirement to write a shell script to search the logs in past 1 hour and extract some pattern from it and count it cumulatively to a file. The problem which I'm facing here is - logs rotates on size basis, say if size of log reaches 5 MB then new log will be generated and... (7 Replies)
Discussion started by: Gem_In_I
7 Replies

4. Shell Programming and Scripting

script to grep outofmemory message in logs

I have prepare script to grep for outofmemory messages in the logs. I need help in modifying script. I have implemented small logic. The outofmemory messages form six logs will store in variables. var1=`grep -i outofmemory $tomcat1logs | sed -n '$p'| sed -n -e "s/.*\(outofmemory\).*/\1/p"`... (6 Replies)
Discussion started by: coolguyamy
6 Replies

5. Shell Programming and Scripting

Concatenate Logs - Perl Question

Hi All, I am fresh to perl and had been using shell scripting in my past experiences. In my part of perl program, i am trying to run a application command ccm stop, which should give some string output as the result. The output (error or sucess) has to be returned to an exisiting log file.... (4 Replies)
Discussion started by: ganga.dharan
4 Replies

6. UNIX for Dummies Questions & Answers

Perl Scripting for monitoring logs

Hi, I am new to perl. I want to write a perl script to monitor logs. Where i want to monitor exceptions logged or any kind of error strings. I have a dir(On Solaris) with multiple log file which keeps rolling to .gz file after some time in that same dir. These logs files size keeps on... (1 Reply)
Discussion started by: solitare123
1 Replies

7. Shell Programming and Scripting

Perl script to rotate logs

I have a shell script that will gzip/tar/archive application logs that are over 20 days old which works just fine, but I would like to convert to a Perl script. Problem is, I'm a beginner with Perl and all attempts so far have failed. Basicaly I have a log dir /app/logs that contains several... (18 Replies)
Discussion started by: theninja
18 Replies

8. UNIX for Advanced & Expert Users

ppp errror message in logs

I am getting this message in the log file. Apr 29 15:32:02 router ppp: Warning: Label COPYRIGHT rejected -direct connection: Configuration label not found This repeats every so often, the link is up however...Any ideas why i am getting this. Its freebsd 6.1 and pppoE. Frank (1 Reply)
Discussion started by: frankkahle
1 Replies

9. UNIX for Dummies Questions & Answers

Message Logs Error

I'm about 5 months new on an 5 year old unix system. If anyone can help me identify what causing the below errors i'd really appreciate it! unix: WARNING: /pci@1f,0/pci@1,1/ide@3/dad@1,0 (dad1): Uncorrectable data Error: Block 57e10 Unix: WARNING: /pci@1f,0/pci@1,1/ide@3/dad@1,0 (dad1):... (1 Reply)
Discussion started by: ByasB
1 Replies
Login or Register to Ask a Question