Perl script to rotate logs


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Perl script to rotate logs
# 1  
Old 03-12-2008
Perl script to rotate logs

I have a shell script that will gzip/tar/archive application logs that are over 20 days old which works just fine, but I would like to convert to a Perl script. Problem is, I'm a beginner with Perl and all attempts so far have failed.
Basicaly I have a log dir /app/logs that contains several different logs files that are created daily. One log for each day.
this.log.2008.02.01
this.log.2008.02.02
this.log.2008.02.03
that.log.2008.02.01
that.log.2008.02.02
that.log.2008.02.03
and so on......

the script finds any log over 20 days old, gzips it, the moves it to /app/logs/archive, after 60 days these files are tar'd up and moved to /log_archive/app_logs

Any help would be greatly appreciated.

here is the shell script.....
Code:
#!/usr/bin/ksh

# Gzip is located in /usr/contrib/bin
export PATH=$PATH:/usr/contrib/bin

# Last 2 digits of the current year
curr_year=`date +%o`

# Directorys
LOG_DIR=/app/logs
AUDIT_DIR=/app/logs/audit
ARCHIVE_DIR=/app/logs/archive
LOG_ARCHIVE_DIR=/log_archive/app_logs

# Number of days before things are gzip'd.
COMPRESS_DAYS=7
# Number of days till gzips are added to a tar file.
MV_AR_ARCHIVE_DAYS=7
# Number of days until the tar files are moved to log partition.
MV_MOAR_ARCHIVE_DAYS=60
# Number of days till tars are removed from log partition.
RM_MOAR_ARCHIVE_DAYS=180

#############

MOAR_HOST_NAME=$(hostname|tr [:lower:] [:upper:])

COMPAREYEAR="`date '+%y'`"
DATEYEAR="`date '+%Y'`"
DATEMONTH="`date '+%m'`"
DATEMONTH=$(( $DATEMONTH - 1 ))

if [ $DATEMONTH -eq 0 ]
then
        DATEMONTH=12
        DATEYEAR=$(( $DATEYEAR - 1 ))
fi
if [ $DATEMONTH -lt 10 ]
then
        DATEMONTH=0$DATEMONTH
fi

# DATESTAMP is the last month
DATESTAMP=$DATEYEAR.$DATEMONTH
# echo $DATESTAMP

#change to the log directory
cd ${LOG_DIR}
echo "Compressing Logs files"
find *fin*log* -prune -mtime +${COMPRESS_DAYS} -exec gzip {} \;
find this* -prune -mtime +${COMPRESS_DAYS} -exec gzip {} \;
find that* -prune -mtime +${COMPRESS_DAYS} -exec gzip {} \;
#move GZIP files to archive directory
find *.gz -mtime +${MV_AR_ARCHIVE_DAYS} -exec mv {} ${LOG_ARCHIVE_DIR}/ \;

#move GZIP files to archive directory
find *.gz -mtime +${MV_AR_ARCHIVE_DAYS} -exec mv {} ${ARCHIVE_DIR}/  \;

#change to the Archive Directory
cd ${ARCHIVE_DIR}
echo "Adding GZIP files to tar archive"
if [ ! -f "arlogs_${DATESTAMP}.tar" ]
then
        touch place.txt
        tar -cf arlogs_${DATESTAMP}.tar place.txt
fi

#loop for all the files in the directory.
ls -l *gz |\
while read file; do
    # Get month
    mo=`echo $file |awk '{print $6}'`
    case $mo in
        Jan) month=01;;
        Feb) month=02;;
        Mar) month=03;;
        Apr) month=04;;
        May) month=05;;
        Jun) month=06;;
        Jul) month=07;;
        Aug) month=08;;
        Sep) month=09;;
        Oct) month=10;;
        Nov) month=11;;
        Dec) month=12;;
          *) echo "Error:$prog:cannot determine month"; exit 1;;
    esac

    # Get day
    day=`echo $file |awk '{print $7}'`
    if [ -z "$day" ]; then
        echo "Error:$prog:cannot determine day"
        exit 1
    fi
    if [ $day -lt 10 ]; then
        day="0$day"
    fi

    # Get year
    file_year=`echo $file |awk '{print $8}'`
    if [ `echo $file_year |grep -c ":"` -gt 0 ]; then
        year=$curr_year
    else
        year=`echo $file_year |sed 's/^..//'`
    fi
    if [ -z "$year" ]; then
        echo "Error:$prog:cannot determine year"
        exit 1
    fi

    # File modification time (timestamp) in xx/xx/xx format
    timestamp="$month/$day/$year"

    # Get filename from long listing
    file=`echo $file |awk '{print $9}'`

    # Add to tar file and remove gzip file
    case $timestamp in
        *$DATESTAMP*)
            tar -uf arlogs_${DATESTAMP}.tar ${file}
            if [ $? -eq 0 ]; then
                rm ${file}
            fi ;;
        $DATEMONTH/*/$COMPAREYEAR)
            tar -uf arlogs_${DATESTAMP}.tar ${file}
            if [ $? -eq 0 ]; then
                rm ${file}
            fi ;;
    esac
done

#check to see if directory on archive partition is there
if [ ! -d "${LOG_ARCHIVE_DIR}/${MOAR_HOST_NAME}_logs_${DATEYEAR}" ]
then
  mkdir ${LOG_ARCHIVE_DIR}/${MOAR_HOST_NAME}_logs_${DATEYEAR}
fi

# Move from the archive directory to the archive partition
find *.tar -mtime +${MV_MOAR_ARCHIVE_DAYS} -exec mv {} ${ARCHIVE_DIR}/${MOAR_HOST_NAME}_logs_${DATEYEAR} \;

#change to the archive partition directory
cd ${LOG_ARCHIVE_DIR}/${MOAR_HOST_NAME}_logs_${DATEYEAR}

# remove any old tars
find *.tar -mtime +${RM_MOAR_ARCHIVE_DAYS} -exec rm {} \;


Thanks
Theninja
# 2  
Old 03-17-2008
anybody, anybody, Bueller, Beuller ?
# 3  
Old 03-17-2008
Quote:
Originally Posted by theninja
anybody, anybody, Bueller, Beuller ?
Posting a working script and basically saying "re-write this for me in Perl" won't work. Maybe that's not how you meant it, but you really didn't give us much to work with.

There are probably no replies because nobody wants to do all your work for you. You say all your attempts have failed. Fair enough, but post your best attempt, or pieces which are failing. I'm sure plenty of people will be willing to help you fix your code.

ShawnMilo
# 4  
Old 03-18-2008
My apologies, I definitley did not mean for someone to re-write it for me, I wouldn't learn anything that way. Here is the test code that is not working for me, it doesn't do anything. No output, no log changes, just sits there until I Ctrl-c out of it.

Code:
#!/usr/local/bin/perl

$LOGPATH='/home/logs';          # Location of AR Log files
$ARCHIVE_DIR='/home/log_archive/moar_logs';     # Location of archived logs
$MONTH=`date +%m`;                      # Current Month
$DAY=`date +%d`;                        # Current Day of month
$MAXCYCLE='2';                          # Number of days to keep log files
$GZIP='/usr/contrib/bin/gzip';          # Location of gzip for compression

@LOGNAMES=('twiz*.log*','nestl*','riesen*');
%ARCHIVE=('*.$MONTH.*'=>1);

chdir $LOGPATH;  # Change to the log directory
foreach $filename (@LOGNAMES) {
  my $oldest = "$filename.$MAXCYCLE";
  archive($oldest) if -e $oldest and $ARCHIVE{$filename};
  for (my $s=$MAXCYCLE; $s >= 0; $s-- ) {
        $oldname = $s ? "$filename.$s" : $filename;
        $newname = join(".",$filename,$s+1);
        rename $oldname,$newname if -e $oldname;
    }
}
kill 'HUP',`cat $PIDFILE`;

printf "$LOGPATH\n";
printf "$ARCHIVE_DIR\n";
printf $MONTH;
printf $DAY;
printf "$MAXDAYS\n";
printf "$GZIP\n"

# 5  
Old 03-18-2008
Code:
@LOGNAMES=('twiz*.log*','nestl*','riesen*');
%ARCHIVE=('*.$MONTH.*'=>1);

Perl doesn't expand the wildcards and metacharacters for you here like the shell would.

It would seem like a good idea to add print statements inside the loop to see where it gets stuck.

Code:
chdir $LOGPATH;  # Change to the log directory
foreach $filename (@LOGNAMES) {
  my $oldest = "$filename.$MAXCYCLE";
print "DEBUG: oldest is '$oldest'\n";
  archive($oldest) if -e $oldest and $ARCHIVE{$filename};
print "DEBUG: '$oldest' exists\n" if -e $oldest;
print "DEBUG: ARCHIVE{'$filename'} is true\n" if $ARCHIVE{$filename};
print "DEBUG: would archive oldest\n" if -e $oldest and $ARCHIVE{$filename};
  for (my $s=$MAXCYCLE; $s >= 0; $s-- ) {
print "DEBUG: s is $s\n";
        $oldname = $s ? "$filename.$s" : $filename;
        $newname = join(".",$filename,$s+1);
print "DEBUG: oldname now '$oldname', newname is '$newname'\n";
print "DEBUG: would rename '$oldname' to '$newname'\n" if -e $oldname;
        rename $oldname,$newname if -e $oldname;
    }
}

I often omit the indentation on debug statements to make it easy to remove them later. A more sustainable solution is to make the prints conditional on a $debug variable, and leave them in.
era
# 6  
Old 03-18-2008
It seems like part of the problem is that the script isn't looking at the files at all. It's looping through strings which you intend to represent filename patterns, but it's never looking at the directory.

Hopefully this will get you started:


Code:
#!/usr/local/bin/perl

opendir(DIR, ".");

@files = grep { /^(twiz*.log*|nestl*|riesen*)/ } readdir(DIR);
closedir(DIR);

foreach $file  (@files){

    print "$file\n";

}

Also, I don't see where the "archive()" function you're calling has been declared. Maybe it's built into your environment. If not, maybe that's what's hanging up.

ShawnMilo
# 7  
Old 03-18-2008
Quote:
Originally Posted by era
Perl doesn't expand the wildcards and metacharacters for you here like the shell would.
Try
Code:
@LOGNAMES=map { glob($_) } 'twiz*.log*','nestl*','riesen*';

Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

If I ran perl script again,old logs should move with today date and new logs should generate.

Appreciate help for the below issue. Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt.............. 1)if logs exist it should move the logs with extention... (1 Reply)
Discussion started by: Sanjeev G
1 Replies

2. UNIX for Dummies Questions & Answers

Logs do not rotate

My problem: Both access and error logs do not rotate any more and get really large. They are located here: /srv/www/+vHost name here+/logs/ Configuration seems to be here: /etc/logrotate.conf => looks OK, including "size 10M" to avoid large files (/etc/logrotate.d => is empty) manually... (4 Replies)
Discussion started by: floko
4 Replies

3. UNIX for Dummies Questions & Answers

Need script to rotate logs

I have few solaris-10 non global zones, where one application is writing some logs to /var/ovd/ConfigLogs. It keeps increasing all the time, as it is needed by application team as of now. I want a small script, which I can configure in cronjob, which should : - Run every Saturday 10 PM - Copy... (5 Replies)
Discussion started by: solaris_1977
5 Replies

4. Shell Programming and Scripting

Perl script to extract last 24 hrs logs from cronlog

Hi Friends, Can anybody help me to create a perl script to generate log file for last 24 hrs from cron log file ?? Thank You (2 Replies)
Discussion started by: syamji.vm
2 Replies

5. Shell Programming and Scripting

Perl script to parse multiple windows event logs.

Hi all, I am developing a log parsing agent in perl to send windows Event logs to Zenoss Monitoring tool. Using Win32::EventLog i can able to get the Event messages but only one Eventype eg Application or System could able to parse at a time. Can you please help to how to open mutiple eventlogs... (3 Replies)
Discussion started by: kar_333
3 Replies

6. Shell Programming and Scripting

Script to rotate file log

Hi Experts, I have script on crontab and give output quite large. I would like to know how to create rotate log when the size of log maximum 50MB if the test.log is 50MB then create test.0 Thanks Edy (2 Replies)
Discussion started by: edydsuranta
2 Replies

7. Shell Programming and Scripting

Help with a rotate log script

Hi all, Am trying to write my own log rotate script. Curremtly, what I have is as below: #!/bin/ksh file_to_rotate=${1} x=${2} while ] do let curr=${x} let prev=${x}-1 if ] ; then #echo "cp -p ${file_to_rotate} ${file_to_rotate}.${curr}" cp -p... (7 Replies)
Discussion started by: newbie_01
7 Replies

8. UNIX for Dummies Questions & Answers

Rotate logs every 1 hour

Hello All, I am learning unix and basically I want to rotate one of my application logs every 1 hour. I need to rotate that file every one hour. I looked in the forums and googled.. but couldn;t get proper information. Requesting you all to kindly guide me. Our application is running on... (4 Replies)
Discussion started by: arunpvp
4 Replies

9. Shell Programming and Scripting

Script for Log Rotate

Hello, I only know the basic for shell programing. I need help for this, I thinks this is a basic for anyone who know a litle of shell scripting. I need creat a script for a rotatate logs, when a filesystem is full. I have a filesystem. The rotate consist in zip the current log (copy) and... (1 Reply)
Discussion started by: El Rengo
1 Replies
Login or Register to Ask a Question