You might use something like the following as a starting point. This script can be used both to rotate log files and to extract unreported entries from old and new log files, to compress old log files (after extracting unreported entries), and to send email containing the extracted unreported entries to email addresses.
This was written and tested using a 1993 version of the Korn shell, but will also work with a recent bash. Obviously, you'll have to adjust variables naming the directory in which your log file is located and the name of your log file. This code assumes that the zipped old log files are to be kept in the same directory. You'll have to make adjustments if you want to move the zipped files to another directory or if you don't like the timestamp I chose as the extension used to name old log files.
Note that the first time you run this script it, it will mail out the entire log file. After that it will keep track of what it reported on the previous run and just mail out entries added since the last run.
Code:
#!/bin/ksh
ec=0 # Final exit code.
IAm="${0##*/}" # Basename of this program.
LOGDIR="/path/to/log/directory" # Directory containing log files.
LOGFILE="syslog" # Name of log file.
LOGPAT="[.][2-9][0-9][0-9][0-9][01][0-9][0-3][0-9]-[0-2][0-9]:[0-5][0-9]:[0-5][0-9]"
STATUSFILE="$LOGFILE.spot" # Status file
TMPF="$IAm.$$" # Temp file to hold extracted log entries.
Usage="SYNOPSIS
$IAm [-hr]" # Synopsis for this program
Help="NAME $IAm -- Rotate and extract recent entries from log files
$Usage
DESCRIPTION
The $IAm utility shall process the log file:
$LOGDIR/$LOGFILE
in various ways. With the -r option, the current log file will be renamed
by appending a timestamp to the end of the log file filename and a new log
file shall be created.
When invoked without options, $IAm shall extract log file entries from
renamed log files and the current log file and mail them to selected
administrators. If any renamed log files are present, $IAm shall zip
those renamed log files after the log entries are extracted.
OPTIONS
-h Help. Print this help message and exit.
-r Rotate log files. If the current log file is not an empty file, the
current log file shall be moved to a file with an extension representing
the current date in the format:
$LOGDIR/$LOGFILE.YYYYMMDD-hh:mm:ss
INPUT FILES
$LOGDIR/$LOGFILE
Current log file.
$LOGDIR/$STATUSFILE
$IAm status file.
$LOGDIR/$LOGFILE.datestamp
Old logfile(s).
OUTPUT FILES
$LOGDIR/$STATUSFILE
Update $IAm status file.
$LOGDIR/$LOGFILE.datestamp.gz
Zipped old logfile(s).
EXIT STATUS
0 Successful commpletion.
>0 An error occurred.
APPLICATION USAGE
Note that this application will only work if each entry written to the log
file is performed as a single write operation and the file descriptor used
to write those entries is opened (for appending) before an entry is written
and closed after each entry is written."
# Move to the directory containing the log and status files..
cd "$LOGDIR" || exit 1
# Process command-line arguments...
while getopts hr name
do case "$name" in
(h) printf '%s\n' "$Help"
exit 0;;
(r) if [ -s "$LOGFILE" ]
then mv "$LOGFILE" "$LOGFILE.$(date '+%Y%m%d-%T')" &&
>> "$LOGFILE" || ec=2
fi
exit $ec;;
(?) printf '%s\n' "$Usage" >&2
exit 3;;
esac
done
shift $((OPTIND - 1))
if [ $# -ne 0 ]
then printf '%s: No operands expected.\n%s\n' "$IAm" "$Usage" >&2
exit 4
fi
# Get last line number processed from the status file...
if [ ! -r "$STATUSFILE" ]
then printf '%s: WARNING: Status file not found. Resetting to line 0.\n' \
"$IAm" >&2
if ! echo 0 > "$STATUSFILE"
then printf "$IAm: Can't create status file. Exiting.\n" "$IAm" >&2
exit 5
fi
last=0
else read -r last < "$STATUSFILE"
fi
# Get list of unzipped log files to process...
list=( "$LOGFILE"$LOGPAT "$LOGFILE" )
if [ "${list[0]}" = "$LOGFILE$LOGPAT" ]
then # There are no old log files, reset the list...
list=( "$LOGFILE" )
fi
# Extract entries added since last run and update status file...
awk -v last="$last" -v StatusFile="$STATUSFILE" '
FNR == 1 && filecount++ {
# We have found the 1st line in a log file after the 1st log file,
# reset "last" so we extract all entries from this log file.
last = 0
}
FNR > last # Extract entries that have not been included in earlier reports.
END { # Update status file.
print FNR > StatusFile
}' "${list[@]}" > "$TMPF" || ec=$((ec + 1))
# Compress any uncompressed old log files...
for ((i = $((${#list[@]} - 2)); i >= 0; i--))
do gzip "${list[$i]}" || ec=$((ec + 1))
done
# Send mail reporting status of this run...
if [ -s "$TMPF" ]
then subject="New log entries found $(date)"
else subject="No new log entries found $(date)"
fi
mailx -s "$subject" user@abc.com user2@xyz.cm < "$TMPF" || ec=$((ec + 1))
rm -f "$TMPF"
exit $ec
give me a shell-script which extract data from log file on a server by giving date and time as input (for both start time and end time) and it will give the logs generated during the given time as output. (4 Replies)
I have to write a script to search the logfiles i.e msg.log for the following
The Search Criteria is as follows
1. IP address
2. String
Ex: abc.123.com
3. Timestamp ( start - end )
ex: 2008-05-04-00:30:00 - 2008-05-08-04:30:00
Can anyone help to devise a script for... (9 Replies)
Hi All,
I would like to write a script which will search a file say abc.dat in /a/b/data for 15 mins only. If the script finds the file in 15 mins then it will exit will exit sucessfully and if there is no file for 15 mins it will exit and copy the last day file (abc.dat_ddmmyyhhmmss) from... (1 Reply)
If I have a log like :
Mon Jul 19 05:07:34 2010; TCP; eth3; 52 bytes; from abc to def
Mon Jul 19 05:07:35 2010; UDP; eth3; 46 bytes; from aaa to bbb
Mon Jul 19 05:07:35 2010; TCP; eth3; 52 bytes; from def to ghi
I will need an output like this :
Time abc to def... (1 Reply)
HI All,
I have a log file where the logs will be in the format as given below:
2011-05-25 02:32:51 INFO PROCESS STARTING
2011-05-25 02:32:52 INFO PROCESS STARTED
.
.
.
I want to retrieve only the logs which are less than 5 mins older than current time using grep... (3 Replies)
I want to grep only last 5 mins of a log file in bash
I have a syslog which contains the following
Mon Jul 11 20:47:42
Mon Jul 11 20:47:52
The following works in Unix but not in AIX . Please can you let me know as to what would be the AIX equivalent
Code: for (( i = 5; i >=0;... (1 Reply)
A happy Monday to you all,
I have a .csv file which contains data taken every 5 seconds. I want to average these 5 second data points into 30 minute averages!
date co2
25/06/2011 08:04 8.31
25/06/2011 08:04 8.32
25/06/2011 08:04 8.33... (18 Replies)
Hello,
Since I m new to shell, I had a hard time to sought out this problem.
I have a log file of a utility which tells that batch files are successful with timestamp. Given below is a part of the log file.
2013/03/07 00:13:50 Apache/1.3.29 (Unix) configured -- resuming normal operations... (12 Replies)
Hello All,
I have to write a script which will search for diffrent patterns like "Struck" "Out of Memory" , etc from a log file in Linux box's.
Now I will be executing a cron job to find out the results by executing the script once in every 30 minutes.
suppose time is
14-04-29:05:31:09
So I... (3 Replies)
Dears,
I am looking for a script which will work as a watch directory.
I ha directory which keep getting files in every 10 mins and some time delay.
I want to monitor if the directory getting the files in every 10 mins if not captured the last received file time and calculate the delay.
... (6 Replies)