Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: How to search backwards in a log file by timestamp of entries?
Top Forums Shell Programming and Scripting How to search backwards in a log file by timestamp of entries? Post 302356711 by ripat on Sunday 27th of September 2009 04:07:41 AM
Old 09-27-2009
For the backwards part, you can use tac. I had some doubts about its efficiency for large files but I just did some tests and, to my great surprise, it is almost as efficient as cat.

Now the parse and time test part. Prerequisite:

- the sample file is exactly as the one you provided. Otherwise you can adjust the field offset by playing around withe the $i's
- you have GNU awk at hand. That's for the systime() and mktime() functions. If not, see remark below.

parselog.awk
Code:
BEGIN{
    FS="[ /:[]"
    now=systime()
    str="Jan_Feb_Mar_Apr_Mai_Jun_Jul_Aug_Sep_Oct_Nov_Dec"
    split(str, m, "_")
    for (i in m) mm[m[i]]=i
}
{
    timestamp=mktime(sprintf("%s %s %s %s %s %s", $7,mm[$6],$5,$8,$9,$10))
    if (timestamp < (now-600)){
        exit
    }
    print
}

To run that snippet:
Code:
$ tac your.log | awk -f parselog.awk

The awk program will stop and exit as soon as it hits a line with a timestamp that is more than 10 min. old. That exit swtich is there to prevent awk to continue scanning the remaining lines which we know will never comply with the timestamp condition.

If you don't have GNU awk, let us know. There is a workaround using awk's system() I/O function and the shell date command.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Spooling a log file with timestamp

Hi From shell script i am invoking sqlplus to connect to oracle database and then i spool a csv file as with output. What i want to do is to change the file name with timestamp on it so after spooling finish shell script change file name with time stamp. can someone help me to do that . Thanks... (2 Replies)
Discussion started by: ukadmin
2 Replies

2. UNIX for Advanced & Expert Users

Copy lines from a log file based on timestamp

how to copy lines from a log file based on timestamp. INFO (RbrProcessFlifoEventSessionEJB.java:processFlight:274) - E_20080521_110754_967: rbrAciInfoObjects listing complete! INFO (RbrPnrProcessEventSessionEJB.java:processFlight:197) - Event Seq: 1647575217; Carrier: UA; Flt#: 0106; Origin:... (1 Reply)
Discussion started by: ranjiadmin
1 Replies

3. Shell Programming and Scripting

Search backwards

Hi, I have a variable , lets say a=/disk1/net/first.ksh i need to grep "first.ksh" everytime "a" gets changed dynamically and i do not know how many '"/" are there in my variable. Can somebody help me out. (9 Replies)
Discussion started by: giri_luck
9 Replies

4. Shell Programming and Scripting

search backwards relative to a string

Hi, I have to search for first occurenceof string str1 in a file(>5GB). Now, after I have that , I have to search backwards from that offset till I get another string str2. I should also be able to get the new string str2's offset. Similarly, I look for last occurence of str1 and then... (1 Reply)
Discussion started by: finder255
1 Replies

5. Shell Programming and Scripting

concatenate log file lines up to timestamp

Hi, Using sed awk or perl I am trying to do something similar to https://www.unix.com/shell-programming-scripting/105887-sed-awk-concatenate-lines-until-blank-line-2.html but my requirement is slightly different. What I am trying to accomplish is to reformat a logfile such that all lines... (4 Replies)
Discussion started by: AlanC
4 Replies

6. Shell Programming and Scripting

search for string and replace backwards

I'm new to Unix scripting and I'm not sure if this can be done. Example: search (grep) in a file for 'Control ID' and then replace with 4 blanks 7 bytes before 'Control ID. input "xxxxxx1234xxxxxxxControl IDxxxxxx" output: "xxxxxx xxxxxxxControl IDxxxxxx" thanks! (7 Replies)
Discussion started by: jbt828
7 Replies

7. Shell Programming and Scripting

Delete log file entries based on the Date/Timestamp within log file

If a log file is in the following format 28-Jul-10 ::: Log message 28-Jul-10 ::: Log message 29-Jul-10 ::: Log message 30-Jul-10 ::: Log message 31-Jul-10 ::: Log message 31-Jul-10 ::: Log message 1-Aug-10 ::: Log message 1-Aug-10 ::: Log message 2-Aug-10 ::: Log message 2-Aug-10 :::... (3 Replies)
Discussion started by: vikram3.r
3 Replies

8. Programming

How to search a file based on a time stamp backwards 10 seconds

Hi all, I'm after some help with this small issue which i'm struggling to work out a fix for. I have a file that contains records that all have a time stamp for each individual record, i need to search the file for a specific time stamp and then search back 10 seconds to see if the number... (2 Replies)
Discussion started by: sp3arsy
2 Replies

9. Shell Programming and Scripting

Search backwards to certain string

Hi, I'm using the following to do a backwards search of a file for a string sed s/^M//g FILE | nawk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r;print;c=a}b{r=$0}' b=10 a=0 s="9005"|grep "policy "|sort -u |awk '{print $4}'|cut -c2-10 My issue is that because I'm looking back 10 lines it's... (11 Replies)
Discussion started by: SaltyDog
11 Replies

10. Shell Programming and Scripting

Search for latest Timestamp in file

Hi, I have a file which generate Timestamp in this format :- 20121012162354 20121114191610 20121210232808 20121216220002 20130106220002 20130127220001 20130203220001 20121012162354 20121114191610 20121210232808 20121216220002 20130106220002 20130127220001 20130203220001 (2 Replies)
Discussion started by: netdbaind
2 Replies

LEARN ABOUT OSX

logsave

LOGSAVE(8)						      System Manager's Manual							LOGSAVE(8)

NAME

       logsave - save the output of a command in a logfile

SYNOPSIS

       logsave [ -asv ] logfile cmd_prog [ ... ]

DESCRIPTION

       The  logsave  program  will  execute  cmd_prog with the specified argument(s), and save a copy of its output to logfile.  If the containing
       directory for logfile does not exist, logsave will accumulate the output in memory until it can be written out.	A copy of the output  will
       also be written to standard output.

       If  cmd_prog  is a single hyphen ('-'), then instead of executing a program, logsave will take its input from standard input and save it in
       logfile

       logsave is useful for saving the output of initial boot scripts until the /var partition is mounted,  so  the  output  can  be  written	to
       /var/log.

OPTIONS

       -a     This option will cause the output to be appended to logfile, instead of replacing its current contents.

       -s     This  option  will  cause  logsave  to  skip writing to the log file text which is bracketed with a control-A (ASCII 001 or Start of
	      Header) and control-B (ASCII 002 or Start of Text).  This allows progress bar information to be visible to the user on the  console,
	      while not being written to the log file.

       -v     This option will make logsave to be more verbose in its output to the user.

AUTHOR

       Theodore Ts'o (tytso@mit.edu)

SEE ALSO

       fsck(8)

E2fsprogs version 1.44.1					    March 2018								LOGSAVE(8)
All times are GMT -4. The time now is 10:21 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy