How to search backwards in a log file by timestamp of entries? Post: 302356711

Sponsored Content

Top Forums Shell Programming and Scripting How to search backwards in a log file by timestamp of entries? Post 302356711 by ripat on Sunday 27th of September 2009 04:07:41 AM

09-27-2009

Registered User

For the backwards part, you can use tac. I had some doubts about its efficiency for large files but I just did some tests and, to my great surprise, it is almost as efficient as cat.

Now the parse and time test part. Prerequisite:

- the sample file is exactly as the one you provided. Otherwise you can adjust the field offset by playing around withe the $i's
- you have GNU awk at hand. That's for the systime() and mktime() functions. If not, see remark below.

parselog.awk

Code:

BEGIN{
    FS="[ /:[]"
    now=systime()
    str="Jan_Feb_Mar_Apr_Mai_Jun_Jul_Aug_Sep_Oct_Nov_Dec"
    split(str, m, "_")
    for (i in m) mm[m[i]]=i
}
{
    timestamp=mktime(sprintf("%s %s %s %s %s %s", $7,mm[$6],$5,$8,$9,$10))
    if (timestamp < (now-600)){
        exit
    }
    print
}

To run that snippet:

Code:

$ tac your.log | awk -f parselog.awk

The awk program will stop and exit as soon as it hits a line with a timestamp that is more than 10 min. old. That exit swtich is there to prevent awk to continue scanning the remaining lines which we know will never comply with the timestamp condition.

If you don't have GNU awk, let us know. There is a workaround using awk's system() I/O function and the shell date command.

ripat

View Public Profile for ripat

Find all posts by ripat

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Spooling a log file with timestamp

Hi From shell script i am invoking sqlplus to connect to oracle database and then i spool a csv file as with output. What i want to do is to change the file name with timestamp on it so after spooling finish shell script change file name with time stamp. can someone help me to do that . Thanks...

2. UNIX for Advanced & Expert Users

Copy lines from a log file based on timestamp

how to copy lines from a log file based on timestamp. INFO (RbrProcessFlifoEventSessionEJB.java:processFlight:274) - E_20080521_110754_967: rbrAciInfoObjects listing complete! INFO (RbrPnrProcessEventSessionEJB.java:processFlight:197) - Event Seq: 1647575217; Carrier: UA; Flt#: 0106; Origin:...

3. Shell Programming and Scripting

Search backwards

Hi, I have a variable , lets say a=/disk1/net/first.ksh i need to grep "first.ksh" everytime "a" gets changed dynamically and i do not know how many '"/" are there in my variable. Can somebody help me out.

4. Shell Programming and Scripting

search backwards relative to a string

Hi, I have to search for first occurenceof string str1 in a file(>5GB). Now, after I have that , I have to search backwards from that offset till I get another string str2. I should also be able to get the new string str2's offset. Similarly, I look for last occurence of str1 and then...

5. Shell Programming and Scripting

concatenate log file lines up to timestamp

Hi, Using sed awk or perl I am trying to do something similar to https://www.unix.com/shell-programming-scripting/105887-sed-awk-concatenate-lines-until-blank-line-2.html but my requirement is slightly different. What I am trying to accomplish is to reformat a logfile such that all lines...

6. Shell Programming and Scripting

search for string and replace backwards

I'm new to Unix scripting and I'm not sure if this can be done. Example: search (grep) in a file for 'Control ID' and then replace with 4 blanks 7 bytes before 'Control ID. input "xxxxxx1234xxxxxxxControl IDxxxxxx" output: "xxxxxx xxxxxxxControl IDxxxxxx" thanks!

7. Shell Programming and Scripting

Delete log file entries based on the Date/Timestamp within log file

If a log file is in the following format 28-Jul-10 ::: Log message 28-Jul-10 ::: Log message 29-Jul-10 ::: Log message 30-Jul-10 ::: Log message 31-Jul-10 ::: Log message 31-Jul-10 ::: Log message 1-Aug-10 ::: Log message 1-Aug-10 ::: Log message 2-Aug-10 ::: Log message 2-Aug-10 :::...

8. Programming

How to search a file based on a time stamp backwards 10 seconds

Hi all, I'm after some help with this small issue which i'm struggling to work out a fix for. I have a file that contains records that all have a time stamp for each individual record, i need to search the file for a specific time stamp and then search back 10 seconds to see if the number...

9. Shell Programming and Scripting

Search backwards to certain string

Hi, I'm using the following to do a backwards search of a file for a string sed s/^M//g FILE | nawk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r;print;c=a}b{r=$0}' b=10 a=0 s="9005"|grep "policy "|sort -u |awk '{print $4}'|cut -c2-10 My issue is that because I'm looking back 10 lines it's...

10. Shell Programming and Scripting

Search for latest Timestamp in file

Hi, I have a file which generate Timestamp in this format :- 20121012162354 20121114191610 20121210232808 20121216220002 20130106220002 20130127220001 20130203220001 20121012162354 20121114191610 20121210232808 20121216220002 20130106220002 20130127220001 20130203220001

LEARN ABOUT CENTOS

igawk

IGAWK(1)							 Utility Commands							  IGAWK(1)

NAME

       igawk - gawk with include files

SYNOPSIS

       igawk [ all gawk options ] -f program-file [ -- ] file ...
       igawk [ all gawk options ] [ -- ] program-text file ...

DESCRIPTION

       Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1).

       AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like

	      @include getopt.awk

       in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path.

OPTIONS

       See gawk(1) for a full description of the AWK language and the options that gawk supports.

EXAMPLES

       cat << EOF > test.awk
       @include getopt.awk

       BEGIN {
	    while (getopt(ARGC, ARGV, "am:q") != -1)
		 ...
       }
       EOF

       igawk -f test.awk

SEE ALSO

       gawk(1)

       Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995.

AUTHOR

       Arnold Robbins (arnold@skeeve.com).

Free Software Foundation					    Nov 3 1999								  IGAWK(1)

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Spooling a log file with timestamp

Discussion started by: ukadmin

2. UNIX for Advanced & Expert Users

Copy lines from a log file based on timestamp

Discussion started by: ranjiadmin

3. Shell Programming and Scripting

Search backwards

Discussion started by: giri_luck

4. Shell Programming and Scripting

search backwards relative to a string

Discussion started by: finder255