Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

To check timestamp in logfile and display lines upto 3 hours before current timestamp

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting To check timestamp in logfile and display lines upto 3 hours before current timestamp
# 1  
Old 07-31-2014
To check timestamp in logfile and display lines upto 3 hours before current timestamp
Hi Friends,

I have the following logfile. Currently time in india is 07/31/2014 12:33:34 and i have the following content in logfile. I want to display only those entries which contain string 'Exception' within last 3 hours. In this case, it would be the last line only

Code:
[7/30/14 4:59:30 Exception Found
[7/30/14 5:18:55 Result is OK
[7/30/14 8:45:22 Exception found
[7/30/14 9:22:26 Exception found
[7/31/14 11:55:66 Exception found


I can get the timestamp in logfile as
Code:
awk '/Exception/ {print $1,$2}' trial.txt | sed 's/^.//' which gives below results

7/30/14 4:59:30
7/30/14 8:45:22
7/30/14 9:22:26
7/31/14 11:55:66

And i can get current timestamp as
Code:
date +"%m/%d/%Y %T"
07/31/2014 12:33:34

So, how can i compare current timestamp with timestamp in log file and display 'Exception' entries until last 3 hours.

---------- Post updated at 02:40 AM ---------- Previous update was at 02:07 AM ----------

I am trying this command. But it is throwing exception saying syntax error. What am i doing wrong?

Code:
awk '/Exception/ if ($1,$2 > $(date +"%m/%d/%Y %T" -d  "3 hour ago")) print $1,$2 ' trial.txt  | sed 's/^.//'

Last edited by Don Cragun; 07-31-2014 at 05:45 AM.. Reason: Add missing CODE tags.
# 2  
Old 07-31-2014
There are several faults:
  • date +%Y gives YYYY, but your logfile has YY
  • $1,$2 won't work. $1" "$2 is a concatenation of 3 strings
  • $(date...) only works inside " " but not ' '
  • A string comparison only works if both the strings have leading zeros
  • A string comparison needs the order "YY MM DD"
Here is an awk solution that reformats with leading zeros and correct order,
and date +%y gives the short year (not Y3k compliant).
Code:
etime=$(date +"%m/%d/%y %T" -d  "3 hour ago")
awk -v etime="$etime" '
BEGIN {
  split(etime,Z)
  split(Z[1],ZD,"/")
  split(Z[2],ZT,":")
  etime=sprintf("%02d/%02d/%02d %02d:%02d:%02d",ZD[3],ZD[2],ZD[1],ZT[1],ZT[2],ZT[3])
}
/Exception/ {
  sub(/^\[/,"")
  split($1,D,"/")
  split($2,T,":")
  time=sprintf("%02d/%02d/%02d %02d:%02d:%02d",D[3],D[2],D[1],T[1],T[2],T[3])
  if (time"" > etime"") {exit}
  print $1,$2
}
' trial.txt

Last edited by MadeInGermany; 07-31-2014 at 07:41 AM.. Reason: ^ added
# 3  
Old 07-31-2014
Hey thanks germany, but the output is showing all the lines with exception. Right now, it should not show any lines.

Code:
$ awk -v etime="$etime" '
> BEGIN {
>   split(etime,Z)
>   split(Z[1],ZD,"/")
>   split(Z[2],ZT,":")
>   etime=sprintf("%02d/%02d/%02d %02d:%02d:%02d",ZD[3],ZD[2],ZD[1],ZT[1],ZT[2],ZT[3])
> }
> /Exception/ {
>   sub(/^\[/,"")
>   split($1,D,"/")
>   split($2,T,":")
>   time=sprintf("%02d/%02d/%02d %02d:%02d:%02d",D[3],D[2],D[1],T[1],T[2],T[3])
>   if (time"" > etime"") {exit}
>   print $1,$2
> }
> ' trial.txt

7/30/14 4:59:30
7/30/14 8:45:22
7/30/14 9:22:26
7/31/14 11:55:66

# 4  
Old 07-31-2014
Well, try this based on MadeInGermany's proposal:
Code:
 awk -vDT=$(date +"%y%m%d%H%M" -d"- 3 hour") '
                 {sub(/^\[/,"")
                  split ($1, D, "/")
                  split ($2, T, ":")
                  AT=sprintf ("%02d%02d%02d%02d%02d", D[3], D[1], D[2], T[1], T[2])}
         AT > DT && /Exception/
        ' file
7/31/14 11:55:66 Exception found

which is fine as my local time is 31.07.14 14:07.
These 2 Users Gave Thanks to RudiC For This Post:
MadeInGermany srkmish
# 5  
Old 07-31-2014
If I'm reading MadeInGermany's code correctly, it is comparing YY/DD/MM HH:MM:SS instead of YY/MM/DD HH:MM:SS and is printing the more than 3 hours ago timestamps instead of the less than 3 hours old timestamps. As long as we're constructing strings to compare, I don't see the need to include the slashes in the dates and I'm also assuming that the minutes and seconds do have 2 digits with zero fill so I don't have to split the time fields (I just have to use leading 0 to fill an 8 character field to supply missing leading zeroes in the hour). And, I used FS instead of sub() and split() to split the date field.

I think RudiC left out a %S in the date format string, but on a 3 hour window, a difference of up to one minute might not be noticeable in the results.

I think this does what was requested (on systems where the date utility supports this form of -d option processing):
Code:
awk -v d="$(date "+%y%m%d%T" -d "3 hours ago")" -F '[[/ ]' '
/Exception/ {
        if(sprintf("%02d%02d%02d%08s", $4, $2, $3, $5) > d)
		printf("%s/%s/%s %s\n", $2, $3, $4, $5)
}' trial.txt

These 2 Users Gave Thanks to Don Cragun For This Post:
MadeInGermany srkmish
# 6  
Old 08-15-2014
Quote:
Originally Posted by RudiC
Well, try this based on MadeInGermany's proposal:
Code:
 awk -vDT=$(date +"%y%m%d%H%M" -d"- 3 hour") '
                 {sub(/^\[/,"")
                  split ($1, D, "/")
                  split ($2, T, ":")
                  AT=sprintf ("%02d%02d%02d%02d%02d", D[3], D[1], D[2], T[1], T[2])}
         AT > DT && /Exception/
        ' file
7/31/14 11:55:66 Exception found

which is fine as my local time is 31.07.14 14:07.
This is perfect, but im not able to store the result by logging into another server. What am i doing wrong?

Code:
 
VAR1=$(ssh server1 "awk -vDT=$(date +"%y%m%d%H%M" -d"- 3 hour") '
                 {sub(/^\[/,"")
                  split ($1, D, "/")
                  split ($2, T, ":")
                  AT=sprintf ("%02d%02d%02d%02d%02d", D[3], D[1], D[2], T[1], T[2])}
         AT > DT && /Exception/
        ' file")

Exception:

Code:
 
awk: cmd. line:1:                  {sub(/^\[/,)
awk: cmd. line:1:                             ^ syntax error
awk: cmd. line:1: fatal: 0 is invalid as number of arguments for sub

# 7  
Old 08-15-2014
Passing complex commands with ssh is problematic, because there are two shells that evaluate the script: one on the local host and one on the remote host.
Save the script on the local host, and pass it via stdin to the remote shell:
Code:
ssh -x server1 "/bin/sh -s" < savedscript

The -s option allows to place script arguments
Code:
ssh -x server1 "/bin/sh -s arg1 arg2" < savedscript

This User Gave Thanks to MadeInGermany For This Post:
srkmish
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep lines between last hour timestamp and current timestamp

So basically I have a log file and each line in this log file starts with a timestamp: MON DD HH:MM:SS SEP 15 07:30:01 I need to grep all the lines between last hour timestamp and current timestamp. Then these lines will be moved to a tmp file from which I will grep for particular strings. ... (1 Reply)
Discussion started by: nms
1 Replies

2. Shell Programming and Scripting

Display lines between timestamp

Hi Gurus, I have a software which logs event in the log file and it has become to big to search into it. I want to display all the lines from the log files between <Jul 21, 2016 3:30:37 PM BST> to <Jul 21, 2016 3:45:37 PM BST> that is 15 min data . Please help Use code tags, thanks. (10 Replies)
Discussion started by: guddu_12
10 Replies

3. Shell Programming and Scripting

AIX : Need to convert UNIX Timestamp to normal timestamp

Hello , I am working on AIX. I have to convert Unix timestamp to normal timestamp. Below is the file. The Unix timestamp will always be preceded by EFFECTIVE_TIME as first field as shown and there could be multiple EFFECTIVE_TIME in the file : 3.txt Contents of... (6 Replies)
Discussion started by: rahul2662
6 Replies

4. HP-UX

Comparing the timestamp of the file to current time

I have a file like this -rwxr-xr-x 1 rewq other 168 Jan 13 07:05 check_files.sh I want to compare (check_files.sh time) with the current time to see if its is older than 2 hours or not if it is not older than 2 hrs then do something.can someone help me on this?.I dont... (7 Replies)
Discussion started by: haadiya
7 Replies

5. Shell Programming and Scripting

Check/Parse log file's lines using time difference/timestamp

I was looking at this script which outputs the two lines which differs less than one sec. #!/usr/bin/perl -w use strict; use warnings; use Time::Local; use constant SEC_MILIC => 1000; my $file='infile'; ## Open for reading argument file. open my $fh, "<", $file or die "Cannot... (1 Reply)
Discussion started by: cele_82
1 Replies

6. Shell Programming and Scripting

Check if a date field has date or timestamp or date&timestamp

Hi, In a field, I should receive the date with time stamp in a particular field. But sometimes the vendor sends just the date or the timestamp or correctl the date&timestamp. I have to figure out the the data is a date or time stamp or date&timestamp. If it is date then append "<space>00:00:00"... (1 Reply)
Discussion started by: machomaddy
1 Replies

7. UNIX for Dummies Questions & Answers

How to compare a file by its timestamp and store in a different location whenever timestamp changes?

Hi All, I am new to unix programming. I am trying for a requirement and the requirement goes like this..... I have a test folder. Which tracks log files. After certain time, the log file is getting overwritten by another file (randomly as the time interval is not periodic). I need to preserve... (2 Replies)
Discussion started by: mailsara
2 Replies

8. Shell Programming and Scripting

How to retrieve the current timestamp?

I am doing this in my script .. currenttimestamp=`db2 "select current timestamp from SYSIBM.SYSDUMMY1 with ur"` echo s $currenttimestamp but this is how its shows s 1 -------------------------- 2011-04-18-12.43.25.345071 1 record(s) selected. How can I just get the timestamp... (6 Replies)
Discussion started by: mitr
6 Replies

9. AIX

Change specific (not current) date to timestamp

Hello to all. I work at AIX system without perl installed and I am restricted user, so I am limited to bash. In script that I am writing, I have to read line from file and transform date that I found inside to Unix timestamp. Line in file look something like this: Tue Mar 29 06:59:00... (5 Replies)
Discussion started by: Hyperborejac
5 Replies

10. Shell Programming and Scripting

Compare current time to timestamp on a file

I'm trying to compare 2 dates between current time and the timestamp on a file. The date format is mmdd Both return Apr 1 but when using if statement line 11: Apr 1: command not found error is returned #!/bin/sh log="DateLog" Current_Date=`date +%b%e` Filepmdate=`ls -l /file.txt |... (1 Reply)
Discussion started by: cillmor
1 Replies
Login or Register to Ask a Question