Shell Programming and Scripting

Any better way to extract the last 10 minutes from logfile?

Code:

  Dec 18 09:41:18 aaa 
Dec 18 09:46:29 aa 
Dec 18 09:48:39 vvv 
Dec 18 09:48:54 bbb 
Dec 18 09:54:47 bbb 
Dec 18 09:55:33 fcf
Dec 18 09:55:38 ssdf 
Dec 18 09:57:58 sdsds 
Dec 18 09:58:10 sdsd 
Dec 18 10:00:50 sdsd 
Dec 18 10:03:43 sdsds 
Dec 18 10:03:50 sds 
Dec 18 10:04:06 sdsd 
Dec 18 10:04:15 sdsd 
Dec 18 10:14:50 sds 
Dec 18 10:19:16 sdsd 
Dec 18 10:19:23 sdsd 
Dec 18 10:21:03 dsdds 
Dec 18 10:22:54 sdsd 
Dec 18 10:27:32 sdsd

Attempt 1: Too slow if the the log file contains over 10000 lines.

Code:

$ lastline=$(tail -n1 logfile) 
$ last10=$(($(date -d "$lastline" +%s) - (10 * 60))) 
$ while read line; do 
> [ $(date -d "${line:0:15}" +%s) -gt $last10 ] && printf "$line\n" 
> done < logfile

Moderator's Comments:

Please use CODE tags. We know that you know how to use CODE tags because you've used them before. But, several of your posts have been edited by moderators to add tags; and several more of your posts should have had tags but nobody bothered to fix them.

---------- Post updated at 06:30 PM ---------- Previous update was at 02:12 PM ----------

Python method is quite fast but only works if the log contains only dates, any suggestion on how to fix this?

Code:

Dec 18 10:19:16
Dec 18 10:19:23
Dec 18 10:21:03
Dec 18 10:22:54
Dec 18 10:27:32

Code:

from datetime import datetime  
astack=[] with open("x.txt") as f:     
for aline in f:         
astack.append(aline.strip()) 
lasttime=datetime.strptime(astack[-1], '%b %d %I:%M:%S') 
for i in astack:     
if (lasttime - datetime.strptime(i, '%b %d %I:%M:%S')).seconds <= 600:         
print i

Real time, or relative to the final post? I assume all important timestamps are formatted identically.

the log will be realtime., Timestamp identical.

C/C++/JAVA/Python or PERL with mmap() and strptime()?

would love a solution either bash, perl or python with fast performance. The bash method mentioned above is a bit too slow.

I wrote a shell tool for time parsing that is already posted here, tm2tm.c However, the shell aproach has you exec'ing for every line to get a comparable time. Something like a tail -r file | sed approach might be able to produce a reverse stream of records with numeric time that can be piped to a shell script, which could then compare and exit for first too low. The time could be an integer munged up from YYYYMMDDhhmmss, and the boundary value can be generated by 'date'. The sed would convert 'Dec 18 10:27:32' to '20141218102732'. The GNU date command or tm2tm can give you that format of 10 minutes ago. A tail -r can be written in C pretty easy using a tmpfile() if input not a flat file, mmap() or some tedious, arduous seek and read and slide in buffer code, or (not good for big files) in shell using an env var or array to hold all the inverted data. I suppose, lacking tail -r, you could just tail off N lines and if not enough, double N and retry (since the file is growing, you have to discard the prior tail data and refetch it with the next tail).