Extract the last 10 minutes from logfile


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract the last 10 minutes from logfile
# 1  
Old 12-26-2013
Extract the last 10 minutes from logfile

Any better way to extract the last 10 minutes from logfile?

Code:
  Dec 18 09:41:18 aaa 
Dec 18 09:46:29 aa 
Dec 18 09:48:39 vvv 
Dec 18 09:48:54 bbb 
Dec 18 09:54:47 bbb 
Dec 18 09:55:33 fcf
Dec 18 09:55:38 ssdf 
Dec 18 09:57:58 sdsds 
Dec 18 09:58:10 sdsd 
Dec 18 10:00:50 sdsd 
Dec 18 10:03:43 sdsds 
Dec 18 10:03:50 sds 
Dec 18 10:04:06 sdsd 
Dec 18 10:04:15 sdsd 
Dec 18 10:14:50 sds 
Dec 18 10:19:16 sdsd 
Dec 18 10:19:23 sdsd 
Dec 18 10:21:03 dsdds 
Dec 18 10:22:54 sdsd 
Dec 18 10:27:32 sdsd

Attempt 1: Too slow if the the log file contains over 10000 lines.

Code:
$ lastline=$(tail -n1 logfile) 
$ last10=$(($(date -d "$lastline" +%s) - (10 * 60))) 
$ while read line; do 
> [ $(date -d "${line:0:15}" +%s) -gt $last10 ] && printf "$line\n" 
> done < logfile

Moderator's Comments:
Mod Comment Please use CODE tags. We know that you know how to use CODE tags because you've used them before. But, several of your posts have been edited by moderators to add tags; and several more of your posts should have had tags but nobody bothered to fix them.


---------- Post updated at 06:30 PM ---------- Previous update was at 02:12 PM ----------

Python method is quite fast but only works if the log contains only dates, any suggestion on how to fix this?

Code:
Dec 18 10:19:16
Dec 18 10:19:23
Dec 18 10:21:03
Dec 18 10:22:54
Dec 18 10:27:32

Code:
from datetime import datetime  
astack=[] with open("x.txt") as f:     
for aline in f:         
astack.append(aline.strip()) 
lasttime=datetime.strptime(astack[-1], '%b %d %I:%M:%S') 
for i in astack:     
if (lasttime - datetime.strptime(i, '%b %d %I:%M:%S')).seconds <= 600:         
print i


Last edited by timmywong; 12-26-2013 at 07:34 AM.. Reason: Add CODE tags.
# 2  
Old 12-26-2013
Real time, or relative to the final post? I assume all important timestamps are formatted identically.
# 3  
Old 12-26-2013
the log will be realtime., Timestamp identical.
# 4  
Old 12-27-2013
C/C++/JAVA/Python or PERL with mmap() and strptime()?
# 5  
Old 12-27-2013
would love a solution either bash, perl or python with fast performance. The bash method mentioned above is a bit too slow.
# 6  
Old 12-27-2013
I wrote a shell tool for time parsing that is already posted here, tm2tm.c However, the shell aproach has you exec'ing for every line to get a comparable time. Something like a tail -r file | sed approach might be able to produce a reverse stream of records with numeric time that can be piped to a shell script, which could then compare and exit for first too low. The time could be an integer munged up from YYYYMMDDhhmmss, and the boundary value can be generated by 'date'. The sed would convert 'Dec 18 10:27:32' to '20141218102732'. The GNU date command or tm2tm can give you that format of 10 minutes ago. A tail -r can be written in C pretty easy using a tmpfile() if input not a flat file, mmap() or some tedious, arduous seek and read and slide in buffer code, or (not good for big files) in shell using an env var or array to hold all the inverted data. I suppose, lacking tail -r, you could just tail off N lines and if not enough, double N and retry (since the file is growing, you have to discard the prior tail data and refetch it with the next tail).

Last edited by DGPickett; 12-27-2013 at 02:43 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep a log file for the last 5 minutes of contents every 5 minutes

Hi all, System Ubuntu 16.04.3 LTS i have the following log INFO 2019-02-07 15:13:31,099 module.py:700] default: "POST /join/8550614e-3e94-4fa5-9ab2-135eefa69c1b HTTP/1.0" 500 2042 INFO 2019-02-07 15:13:31,569 module.py:700] default: "POST /join/6cb9c452-dcb1-45f3-bcca-e33f5d450105... (15 Replies)
Discussion started by: charli1
15 Replies

2. UNIX for Beginners Questions & Answers

How to convert days hours minutes seconds to minutes?

Hi, please help with below time conversion to minutes. one column values: 2 minutes 16 seconds 420 msec 43 seconds 750 msec 0 days 3 hours 29 minutes 58 seconds 480 msec 11 seconds 150 msec I need output in minutes(total elapsed time in minutes) (2 Replies)
Discussion started by: ramu.badugula
2 Replies

3. Shell Programming and Scripting

How to extract logs between the current time and the last 15 minutes ?

I want to extract the logs between the current time stamp and 15 minutes before and sent an email to the people configured. I developed the below script but it's not working properly; can someone help me?? I have a log file containing this pattern: Constructor QuartzJob ... (3 Replies)
Discussion started by: puneetkhullar
3 Replies

4. Shell Programming and Scripting

Logfile monitoring with logfile replacement

Bonjour, I've wrote a script to monitor a logfile in realtime. It is working almost perfeclty except for two things. The script use the following technique : tail -fn0 $logfile | \ while read line ; do ... some stuff done First one, I'd like a way to end the monitoring script if a... (3 Replies)
Discussion started by: Warluck
3 Replies

5. Shell Programming and Scripting

Checking if logfile updates after 5 minutes

I would like to create a script that will send an email to me if it does not update after 5 minutes. I have created this script but it doesn't work. #!/bin/ksh # Variable Declaration log=/opt/app/netcool/omnibus/log.txt servertime=`date | awk '{print $2,$3,$4 }' | cut -d ":" -f1,2`... (5 Replies)
Discussion started by: Dan Llave
5 Replies

6. Shell Programming and Scripting

Extract three substrings from a logfile

I have a log file like below. 66.249.73.11 - - "UCiZ7QocVqYAABgwfP8AAHAA" "US" "Mediapartners-Google" "-" www.mahashwetha.com.sg "GET... (2 Replies)
Discussion started by: Tuxidow
2 Replies

7. Shell Programming and Scripting

Perl - Extract 12 hour time, convert to 24 and subtract 15 minutes?

OK, I am by no means a programmer... I have been given the task to do some automation scripts. I have got most of it working from snippets I have found on the Web. One requirement has me stumped. The initial timing file created by the user is a comma delimited in the following format.... (4 Replies)
Discussion started by: autotuner
4 Replies

8. Shell Programming and Scripting

Script to extract line from logfile

Hi , Can someone help me,I am not well versed with scripting,it is very urjent. Need a script in perl or shell for the following task. The logfile contains several entries of which one particular string to be searched and that complete line has to be removed and copied to another file to... (25 Replies)
Discussion started by: garryt21@rediff
25 Replies

9. Shell Programming and Scripting

Need to extract specific pattern from logfile

Log File: Attempting to contact (DESCRIPTION=(SOURCE_ROUTE=OFF)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=hostname1.com)(PORT=1521)))(CONNECT_DATA=(SID=database1)(SRVR=DEDICATED))) Attempting to contact... (2 Replies)
Discussion started by: techychap
2 Replies

10. Shell Programming and Scripting

Convert minutes to hours, minutes, seconds

How would you convert lets say a 1000 minutes to hours, minutes, seconds (1 Reply)
Discussion started by: Vozx
1 Replies
Login or Register to Ask a Question