Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Script to search log file for last 15 mins data Post 302956794 by methyl on Saturday 3rd of October 2015 08:09:52 PM
Old 10-03-2015
Quote:
I strongly suggest that instead of trying to match based on timestamps you instead keep track of the line number of the last line processed in the previous run and on the next run just start processing with the next line in that log file. Doing it this way will keep you from missing events and keep you from processing some events twice.
Furthur to Don_Cragun, and if I have understood the task correctly.

You need a script running under cron every 15 minutes which copies the current logfile and compares the line count with the previous version of the logfile. The script then processes the "tail -#" number of lines difference.

This is a basic technique to avoid "tail -f" on logfiles which will get deleted by log rotation scripts. It also avoids reading the logfile and processing all the timestamps in the logfile.

Don't forget the "first time" condition when the T-15 minutes copy does not exist. Your first run of the cron will just prepare the files.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

shell-script which extract data from log file

give me a shell-script which extract data from log file on a server by giving date and time as input (for both start time and end time) and it will give the logs generated during the given time as output. (4 Replies)
Discussion started by: abhishek27
4 Replies

2. Shell Programming and Scripting

{How} Script to search a log file for a given criteria

I have to write a script to search the logfiles i.e msg.log for the following The Search Criteria is as follows 1. IP address 2. String Ex: abc.123.com 3. Timestamp ( start - end ) ex: 2008-05-04-00:30:00 - 2008-05-08-04:30:00 Can anyone help to devise a script for... (9 Replies)
Discussion started by: indiakingz
9 Replies

3. Shell Programming and Scripting

Script which will search for a file for 15 mins

Hi All, I would like to write a script which will search a file say abc.dat in /a/b/data for 15 mins only. If the script finds the file in 15 mins then it will exit will exit sucessfully and if there is no file for 15 mins it will exit and copy the last day file (abc.dat_ddmmyyhhmmss) from... (1 Reply)
Discussion started by: chandancsc
1 Replies

4. Shell Programming and Scripting

need a shell script to extract data from a log file.

If I have a log like : Mon Jul 19 05:07:34 2010; TCP; eth3; 52 bytes; from abc to def Mon Jul 19 05:07:35 2010; UDP; eth3; 46 bytes; from aaa to bbb Mon Jul 19 05:07:35 2010; TCP; eth3; 52 bytes; from def to ghi I will need an output like this : Time abc to def... (1 Reply)
Discussion started by: hitha87
1 Replies

5. Shell Programming and Scripting

Retrieve logs generated in last 10 mins from a log file using 'grep' command

HI All, I have a log file where the logs will be in the format as given below: 2011-05-25 02:32:51 INFO PROCESS STARTING 2011-05-25 02:32:52 INFO PROCESS STARTED . . . I want to retrieve only the logs which are less than 5 mins older than current time using grep... (3 Replies)
Discussion started by: rvhg16
3 Replies

6. AIX

Grep last 5 mins from log file in AIX

I want to grep only last 5 mins of a log file in bash I have a syslog which contains the following Mon Jul 11 20:47:42 Mon Jul 11 20:47:52 The following works in Unix but not in AIX . Please can you let me know as to what would be the AIX equivalent Code: for (( i = 5; i >=0;... (1 Reply)
Discussion started by: necro98
1 Replies

7. Shell Programming and Scripting

Averaging data every 30 mins using AWK

A happy Monday to you all, I have a .csv file which contains data taken every 5 seconds. I want to average these 5 second data points into 30 minute averages! date co2 25/06/2011 08:04 8.31 25/06/2011 08:04 8.32 25/06/2011 08:04 8.33... (18 Replies)
Discussion started by: gd9629
18 Replies

8. Shell Programming and Scripting

Script (ksh) to get data in every 30 mins interval for the given date

Hello, Since I m new to shell, I had a hard time to sought out this problem. I have a log file of a utility which tells that batch files are successful with timestamp. Given below is a part of the log file. 2013/03/07 00:13:50 Apache/1.3.29 (Unix) configured -- resuming normal operations... (12 Replies)
Discussion started by: rpm120
12 Replies

9. Shell Programming and Scripting

Script to search for a pattern in 30 minutes from a log file

Hello All, I have to write a script which will search for diffrent patterns like "Struck" "Out of Memory" , etc from a log file in Linux box's. Now I will be executing a cron job to find out the results by executing the script once in every 30 minutes. suppose time is 14-04-29:05:31:09 So I... (3 Replies)
Discussion started by: Shubhasis Mathr
3 Replies

10. Shell Programming and Scripting

Script to find directory is getting files in every 10 mins, if not then when last time file received

Dears, I am looking for a script which will work as a watch directory. I ha directory which keep getting files in every 10 mins and some time delay. I want to monitor if the directory getting the files in every 10 mins if not captured the last received file time and calculate the delay. ... (6 Replies)
Discussion started by: sadique.manzar
6 Replies
ROTATELOGS(8)							    rotatelogs							     ROTATELOGS(8)

NAME
rotatelogs - Piped logging program to rotate Apache logs SYNOPSIS
rotatelogs [ -l ] [ -L linkname ] [ -p program ] [ -f ] [ -v ] [ -e ] [ -c ] [ -n number-of-files ] logfile rotationtime|filesize(B|K|M|G) [ offset ] SUMMARY
rotatelogs is a simple program for use in conjunction with Apache's piped logfile feature. It supports rotation based on a time interval or maximum size of the log. OPTIONS
-l Causes the use of local time rather than GMT as the base for the interval or for strftime(3) formatting with size-based rotation. -L linkname Causes a hard link to be made from the current logfile to the specified link name. This can be used to watch the log continuously across rotations using a command like tail -F linkname. -p program If given, rotatelogs will execute the specified program every time a new log file is opened. The filename of the newly opened file is passed as the first argument to the program. If executing after a rotation, the old log file is passed as the second argument. rotatelogs does not wait for the specified program to terminate before continuing to operate, and will not log any error code returned on termination. The spawned program uses the same stdin, stdout, and stderr as rotatelogs itself, and also inherits the environment. -f Causes the logfile to be opened immediately, as soon as rotatelogs starts, instead of waiting for the first logfile entry to be read (for non-busy sites, there may be a substantial delay between when the server is started and when the first request is handled, meaning that the associated logfile does not "exist" until then, which causes problems from some automated logging tools) -t Causes the logfile to be truncated instead of rotated. This is useful when a log is processed in real time by a command like tail, and there is no need for archived data. No suffix will be added to the filename, however format strings containing '%' characters will be respected. -v Produce verbose output on STDERR. The output contains the result of the configuration parsing, and all file open and close actions. -e Echo logs through to stdout. Useful when logs need to be further processed in real time by a further tool in the chain. -c Create log file for each interval, even if empty. -n number-of-files Use a circular list of filenames without timestamps. With -n 3, the series of log files opened would be "logfile", "logfile.1", "logfile.2", then overwriting "logfile". logfile rotationtime The time between log file rotations in seconds. The rotation occurs at the beginning of this interval. For example, if the rotation time is 3600, the log file will be rotated at the beginning of every hour; if the rotation time is 86400, the log file will be rotated every night at midnight. (If no data is logged during an interval, no file will be created.) filesize(B|K|M|G) The maximum file size in followed by exactly one of the letters B (Bytes), K (KBytes), M (MBytes) or G (GBytes). .PP When time and size are specified, the size must be given after the time. Rotation will occur whenever either time or size limits are reached. offset The number of minutes offset from UTC. If omitted, zero is assumed and UTC is used. For example, to use local time in the zone UTC -5 hours, specify a value of -300 for this argument. In most cases, -l should be used instead of specifying an offset. EXAMPLES
CustomLog "|bin/rotatelogs /var/logs/logfile 86400" common This creates the files /var/logs/logfile.nnnn where nnnn is the system time at which the log nominally starts (this time will always be a multiple of the rotation time, so you can synchronize cron scripts with it). At the end of each rotation time (here after 24 hours) a new log is started. CustomLog "|bin/rotatelogs -l /var/logs/logfile.%Y.%m.%d 86400" common This creates the files /var/logs/logfile.yyyy.mm.dd where yyyy is the year, mm is the month, and dd is the day of the month. Logging will switch to a new file every day at midnight, local time. CustomLog "|bin/rotatelogs /var/logs/logfile 5M" common This configuration will rotate the logfile whenever it reaches a size of 5 megabytes. ErrorLog "|bin/rotatelogs /var/logs/errorlog.%Y-%m-%d-%H_%M_%S 5M" This configuration will rotate the error logfile whenever it reaches a size of 5 megabytes, and the suffix to the logfile name will be cre- ated of the form errorlog.YYYY-mm-dd-HH_MM_SS. CustomLog "|bin/rotatelogs -t /var/logs/logfile 86400" common This creates the file /var/logs/logfile, truncating the file at startup and then truncating the file once per day. It is expected in this scenario that a separate process (such as tail) would process the file in real time. PORTABILITY
The following logfile format string substitutions should be supported by all strftime(3) implementations, see the strftime(3) man page for library-specific extensions. o %A - full weekday name (localized) o %a - 3-character weekday name (localized) o %B - full month name (localized) o %b - 3-character month name (localized) o %c - date and time (localized) o %d - 2-digit day of month o %H - 2-digit hour (24 hour clock) o %I - 2-digit hour (12 hour clock) o %j - 3-digit day of year o %M - 2-digit minute o %m - 2-digit month o %p - am/pm of 12 hour clock (localized) o %S - 2-digit second o %U - 2-digit week of year (Sunday first day of week) o %W - 2-digit week of year (Monday first day of week) o %w - 1-digit weekday (Sunday first day of week) o %X - time (localized) o %x - date (localized) o %Y - 4-digit year o %y - 2-digit year o %Z - time zone name o %% - literal `%' Apache HTTP Server 2013-06-13 ROTATELOGS(8)
All times are GMT -4. The time now is 09:50 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy