Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Check/Parse log file's lines using time difference/timestamp
Top Forums Shell Programming and Scripting Check/Parse log file's lines using time difference/timestamp Post 302858757 by cele_82 on Tuesday 1st of October 2013 10:31:44 AM
Old 10-01-2013
Check/Parse log file's lines using time difference/timestamp
I was looking at this script which outputs the two lines which differs less than one sec.



Code:
#!/usr/bin/perl -w

use strict;
use warnings;
use Time::Local;
use constant SEC_MILIC => 1000;

my $file='infile';

## Open for reading argument file.
open my $fh, "<", $file or die "Cannot open file $file $!\n";

## Save previous values.
my ($time_prev, $reg_prev);

while ( <$fh> ) { 
    ## Get Year/Month/Day/hour/minute/second/milisecond from input line.
    /^\s*(\d{4})-(\d{2})-(\d{2})\s+(\d{2}):(\d{2}):(\d{2}),(\d{3})/;
    ## Get number of seconds form Epoch.
    my $time = timelocal( $6, $5, $4, $3, $2 - 1, $1 );
    ## Update to miliseconds.
    $time += $7 / SEC_MILIC;


    ## Cannot compare times in first line of file, save values and read next.
    if ($. == 1) {
        $time_prev = $time;
        $reg_prev = $_; 
        next;
    }   

    ## Check difference less than a second and print both lines.
    if ( 1 > abs($time - $time_prev) ) { 
        print "$reg_prev", "$_", "\n";
    }   

    ## Save current values to compare with next line.
    $time_prev = $time;
    $reg_prev = $_; 
}

Which works ok if the logs are formatted like this.

Code:
2011-02-04 11:11:12,923 Message to msisdn: XXXXXXXXXXXX
2011-02-04 11:11:14,950 Message to msisdn: XXXXXXXXXXXX
2011-02-04 11:11:16,967 Message to msisdn: XXXXXXXXXXXX
2011-02-04 11:11:18,982 Message to msisdn: XXXXXXXXXXXX
2011-02-04 11:11:19,499 Message to msisdn: XXXXXXXXXXXX

---------- Post updated at 03:31 PM ---------- Previous update was at 03:27 PM ----------

Can you suggest how to adapt the script to parse another logs but formatted in different way?

Code:
Sep 30 21:08:00 error segfault 0x0002220ff blah blah blah
Sep 30 21:09:00 read this read that
Sep 30 21:10:00 done this done that

I would like to get an output when the time difference of two following lines is greater than 1hr in time.


If you have a look it would be much appreciated


Thanks
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

To find the time difference between two lines of the same log file

Hello Friends, I want to write a script for the following: nlscux62:tibprod> grep "2008 Apr 30 01:" SA_EHV_SPEED_SFC_IN_03-SA_EHV_SPEED_SFC_IN_03-2.log | grep -i post | more 2008 Apr 30 01:01:23:928 GMT +2 SAPAdapter.SA_EHV_SPEED_SFC_IN_03-SA_EHV_SPEED_SFC_IN_03-2 Info AER3-000095 IDOC... (2 Replies)
Discussion started by: satyakam
2 Replies

2. UNIX for Advanced & Expert Users

Copy lines from a log file based on timestamp

how to copy lines from a log file based on timestamp. INFO (RbrProcessFlifoEventSessionEJB.java:processFlight:274) - E_20080521_110754_967: rbrAciInfoObjects listing complete! INFO (RbrPnrProcessEventSessionEJB.java:processFlight:197) - Event Seq: 1647575217; Carrier: UA; Flt#: 0106; Origin:... (1 Reply)
Discussion started by: ranjiadmin
1 Replies

3. Shell Programming and Scripting

concatenate log file lines up to timestamp

Hi, Using sed awk or perl I am trying to do something similar to https://www.unix.com/shell-programming-scripting/105887-sed-awk-concatenate-lines-until-blank-line-2.html but my requirement is slightly different. What I am trying to accomplish is to reformat a logfile such that all lines... (4 Replies)
Discussion started by: AlanC
4 Replies

4. Shell Programming and Scripting

Need to parse file "x" lines at a time ... awk array?

I have files that store multiple data points for the same device "vertically" and include multiple devices. It repeats a consistant pattern of lines where for each line: Column 1 is a common number for the entire file and all devices in that file Column 2 is a unique device number Column 3 is... (7 Replies)
Discussion started by: STN
7 Replies

5. Shell Programming and Scripting

Find time difference between two consecutive lines in same file.

Hello I have a file in following format: IV 08:09:07 NM 08:12:01 IC 08:12:00 MN 08:14:20 NM 08:14:15 I need a script to compare time on each line with previous line and show the inconsecutive line. Ex.: 08:12:00 08:14:15 A better way... (6 Replies)
Discussion started by: vilibit
6 Replies

6. Shell Programming and Scripting

Transpose timestamp based on column values and calculate time difference

Hello Expert, I need to transpose Date-Timestamp based on same column values and calculate time difference. The input file would be as below and required output is mentioned in the bottom INPUT File ======== 08/23/2012 12:36:09 JOB_5340 08/23/2012 12:36:14 JOB_5340 08/23/2012... (2 Replies)
Discussion started by: asnandhakumar
2 Replies

7. UNIX for Dummies Questions & Answers

ksh to check second time difference between two servers

I am currently setting up a public key authentication between servers. The goal is to get the date via `ssh hostname date` on all the 4 remote servers , put the value in a text file on the central server and compare the date (specifically seconds) for each server date output to check if time is... (7 Replies)
Discussion started by: depam
7 Replies

8. Shell Programming and Scripting

To check timestamp in logfile and display lines upto 3 hours before current timestamp

Hi Friends, I have the following logfile. Currently time in india is 07/31/2014 12:33:34 and i have the following content in logfile. I want to display only those entries which contain string 'Exception' within last 3 hours. In this case, it would be the last line only I can get the... (12 Replies)
Discussion started by: srkmish
12 Replies

9. Shell Programming and Scripting

Shell Script | Parse log file after a given date and time stamp

I am developing one script which will take log file name, output file name, date, hour and minute as an argument and based on these inputs, the script will scan and capture all the error(s) that have been triggered from a given time. Example: script should capture all the error after 13:50 on Jan... (2 Replies)
Discussion started by: ROMA3
2 Replies

10. UNIX for Beginners Questions & Answers

Parse apache log file with three different time formats

Hi, I want to parse below file and Write a function to extract the logs between two given timestamp. Apache (Unix) Log Samples - MonitorWare The challenge here is there are three date and time format. First :- 07/Mar/2004:16:05:49 Second :- Sun Mar 7 16:02:00 2004 Third :- 29-Mar... (6 Replies)
Discussion started by: sahil_shine
6 Replies

LEARN ABOUT REDHAT

checkbashisms

CHECKBASHISMS(1)					      General Commands Manual						  CHECKBASHISMS(1)

NAME

       checkbashisms - check for bashisms in /bin/sh scripts

SYNOPSIS

       checkbashisms script ...
       checkbashisms --help|--version

DESCRIPTION

       checkbashisms, based on one of the checks from the lintian system, performs basic checks on /bin/sh shell scripts for the possible presence
       of bashisms.  It takes the names of the shell scripts on the command line, and outputs warnings if possible bashisms are detected.

       Note that the definition of a bashism in this context roughly equates to "a shell feature that is not required to be supported  by  POSIX";
       this means that some issues flagged may be permitted under optional sections of POSIX, such as XSI or User Portability.

       In  cases  where  POSIX	and  Debian  Policy  disagree, checkbashisms by default allows extensions permitted by Policy but may also provide
       options for stricter checking.

OPTIONS

       --help, -h
	      Show a summary of options.

       --newline, -n
	      Check for "echo -n" usage (non POSIX but required by Debian Policy 10.4.)

       --posix, -p
	      Check for issues which are non POSIX but required to be supported by Debian Policy 10.4 (implies -n).

       --force, -f
	      Force each script to be checked, even if it would normally not be (for instance, it has a bash or non POSIX shell shebang or appears
	      to be a shell wrapper).

       --extra, -x
	      Highlight lines which, whilst they do not contain bashisms, may be useful in determining whether a particular issue is a false posi-
	      tive which may be ignored.  For example, the use of "$BASH_ENV" may be preceded by checking whether "$BASH" is set.

       --version, -v
	      Show version and copyright information.

EXIT VALUES

       The exit value will be 0 if no possible bashisms or other problems were detected.  Otherwise it will be the sum of the following error val-
       ues:

       1      A possible bashism was detected.

       2      A file was skipped for some reason, for example, because it was unreadable or not found.	The warning message will give details.

SEE ALSO

       lintian(1).

AUTHOR

       checkbashisms  was originally written as a shell script by Yann Dirson <dirson@debian.org> and rewritten in Perl with many more features by
       Julian Gilbey <jdg@debian.org>.

DEBIAN
								 Debian Utilities						  CHECKBASHISMS(1)
All times are GMT -4. The time now is 09:24 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy