Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Grep a log file for the last 5 minutes of contents every 5 minutes

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Grep a log file for the last 5 minutes of contents every 5 minutes
# 1  
Old 02-08-2019
Grep a log file for the last 5 minutes of contents every 5 minutes
Hi all,
System Ubuntu 16.04.3 LTS
i have the following log
Code:
INFO     2019-02-07 15:13:31,099 module.py:700] default: "POST /join/8550614e-3e94-4fa5-9ab2-135eefa69c1b HTTP/1.0" 500 2042
INFO     2019-02-07 15:13:31,569 module.py:700] default: "POST /join/6cb9c452-dcb1-45f3-bcca-e33f5d450105 HTTP/1.0" 500 2042
INFO     2019-02-07 15:14:25,379 module.py:700] default: "POST /join/f30c767d-fcd2-44c2-83a9-d927fv10f7a8 HTTP/1.0" 500 2042
INFO     2019-02-07 15:14:46,531 module.py:700] default: "POST /join/57e7930c-99ea-419a-b807-fad6327cd6e9 HTTP/1.0" 500 2042
INFO     2019-02-07 15:15:06,966 module.py:700] default: "POST /join/57e7930c-99ea-419a-b807-fad6447cd6e9 HTTP/1.0" 500 2042
INFO     2019-02-07 15:15:06,966 module.py:700] default: "POST /join/57e793rfc-98ea-418a-b817-fad6347cd6e9 HTTP/1.0" 500 2042

i need to create a monitor for this log and grep only last 5 minutes of this log HTTP/1.0" 500 2042, and make an action if the match count is higher then 5 in last 5 minutes.

I tyied the following script but for some reason i can't get it working


Code:
#!/bin/bash
#
check=$(awk -v d1="$(date --date="-5 min" "+%Y-%m-%d %H:%M:%S,%3N")" -v d2="$(date "+%Y-%m-%d %H:%M:%S,%3N")" '$0 > d1 && $0 < d2 || $0 ~ d2' /var/log/messages | grep -i "HTTP/1.0" 500 2042")
count=$(awk -v d1="$(date --date="-5 min" "+%Y-%m-%d %H:%M:%S,%3N")" -v d2="$(date "+%Y-%m-%d %H:%M:%S,%3N")" '$0 > d1 && $0 < d2 || $0 ~ d2' /var/log/messages | grep -ci "HTTP/1.0" 500 2042") 
if [ $count -gt 5 ] 
then
   echo $CHECK | /bin/mail -s "$count occurrences of the error message has been found in the last 5 minutes" myemail@mydomain.com 
else
   echo "Exit, everything is ok" > /dev/null
fi

The error i gues is with grep because there are the " in the middle.
Could somebody help me on this what i'm i missing here?
Thank you in advance.
Last edited by charli1; 02-08-2019 at 08:06 AM..
# 2  
Old 02-08-2019
I do not know how it should work without a plus sign ?
Code:
date --date="-5 min" "%Y-%m-%d %H:%M:%S,%3N"

Last edited by vbe; 02-08-2019 at 08:26 AM.. Reason: code tags not icode please, thanks
# 3  
Old 02-08-2019
Yes, sorry i wrote this without the +, this is a typo i made here, but the original copy has is, and it doesn't work either.
Thank you.
Last edited by vbe; 02-08-2019 at 08:27 AM.. Reason: spelling
This User Gave Thanks to charli1 For This Post:
nezabudka
# 4  
Old 02-08-2019
Here to begin with, and I will try to understand
Code:
awk -F, -v d="$(date --date="-5 minutes" +"%Y-%m-%d %H:%M:%S,%3N")" '( d > substr($1, 10)) { print }' file

--- Post updated at 12:38 ---

Just need to escape quotes
Code:
grep -i "HTTP/1.0\" 500 2042"

--- Post updated at 12:45 ---

And rearrange options. Like this
grep -iс

--- Post updated at 13:03 ---

the awk itself well knows how to count
Code:
awk -F, -v d="$(date --date="-50 minutes" +"%Y-%m-%d %H:%M:%S,%3N")"
'$0 ~ "HTTP/1.0\" 500 2042$" {if ( d > substr($1, 10)) error++} END {print error}' file

Last edited by nezabudka; 02-08-2019 at 09:15 AM..
# 5  
Old 02-08-2019
Quote:
Originally Posted by nezabudka

Just need to escape quotes
Code:
grep -i "HTTP/1.0\" 500 2042"

or use different quotes, to avoid ugly escaping...

Code:
grep -i 'HTTP/1.0" 500 2042'

I would omit the fixed request size in the grep too, maybe that changes at any time(software upgrade?)
Code:
grep -i 'HTTP/1.0" 500 [0-9]+'

This User Gave Thanks to stomp For This Post:
nezabudka
# 6  
Old 02-08-2019
Hello nezabudka,
this code is printing all the matchs on the file, and not only the last five minutes of the file,
any idea?
Code:
awk -F, -v d="$(date --date="-5 minutes" +"%Y-%m-%d %H:%M:%S,%3N")"
'$0 ~ "HTTP/1.0\" 500 2042$" {if ( d > substr($1, 10)) error++} END {print error}' file

This User Gave Thanks to charli1 For This Post:
nezabudka
# 7  
Old 02-08-2019
I was mistaken
Code:
( d > substr($1, 10))

true
Code:
( d < substr($1, 10))
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Copy last 30 minutes' contents from a log file

Hi Guys, I am writing a bash script to capture the last 30 minutes's contents from log file to a new file. This job is a scheduled job and will run every 30 minutes. The log file is db2diag.log in DB2. I am having difficulties copying the last 30 minutes's contents. Can someone please help me.... (4 Replies)
Discussion started by: naveed
4 Replies

2. Shell Programming and Scripting

Check file creation Time minutes and if file older then 5 minutes execute some stuff

Hello all, Info: System RedHat 7.5 I need to create a script that based on the creation time, if the file is older then 5 minutes then execute some stuff, if not exit. I thought to get the creation time and minutes like this. CreationTime=$(stat -c %y /tmp/test.log | awk -F" " '{ print... (3 Replies)
Discussion started by: charli1
3 Replies

3. UNIX for Beginners Questions & Answers

How to convert days hours minutes seconds to minutes?

Hi, please help with below time conversion to minutes. one column values: 2 minutes 16 seconds 420 msec 43 seconds 750 msec 0 days 3 hours 29 minutes 58 seconds 480 msec 11 seconds 150 msec I need output in minutes(total elapsed time in minutes) (2 Replies)
Discussion started by: ramu.badugula
2 Replies

4. Shell Programming and Scripting

Grep last 2 minutes log only

I have newbie, which i use for checking last one hours log file, but i want to check 2 minutes log and discard old log, only match current time with last 2 minutes. Ex log. 2018-07-03 20:09:17 2018-07-03 20:05:17 2018-07-03 20:05:18 2018-07-03 20:05:20 2018-07-03 20:06:22 2018-07-03... (5 Replies)
Discussion started by: ooilinlove
5 Replies

5. Shell Programming and Scripting

Need help in getting the Last 30 minutes logs from the Log File

I have a log file with the below contents : log_file_updated.txt : Jul 5 03:33:06 rsyslogd: was Jul 5 03:33:09 adcsdb1 rhsmd: This system is registered. Sep 2 02:45:48 adcsdb1 UDSAgent: 2015-07-05 04:24:48.959 INFO Worker_Thread_4032813936 Accepted connection from host <unknown>... (3 Replies)
Discussion started by: rahul2662
3 Replies

6. Shell Programming and Scripting

Grep last 30 minutes log only

I have below command, which i use for checking last two hours log file, but i want to check 30 minutes log and discard old log, only match current time with last 30 minutes. Command i am using. This below attach log file format is for this week, but sometime it got change in time of log, so i need... (6 Replies)
Discussion started by: learnbash
6 Replies

7. Shell Programming and Scripting

Log File Creations for every 60 minutes

Hi All, Below script will make a copy of the existing log file with the then timestamp details. I am looking to create a copy of the existing log file for every 60 minutes and when the file limit reaches to 5, the 6th copy should overwrite the first backedup file which means all the time it... (3 Replies)
Discussion started by: Upendra Bestha
3 Replies

8. Shell Programming and Scripting

Script to grep for a string in log files generated in last 15 minutes.

Dear Guru's I've a requirment to grep for a string in series of log files that are getting generated almost every minute. I'm looking to schedule a script every 15 mountes,in order to check if the error string has been generated in any of the log files generated in last 15 minutes. Please... (3 Replies)
Discussion started by: rajivatnova
3 Replies

9. Shell Programming and Scripting

Grepping the last 30 minutes of a log file...

I need to know if anyone can assist me on how to grab the last (we'll just say "x" minutes) of a log file. How do you tell the grep command without specifying an exact window of time? (So relative instead of absolute.) Thanks, Jon (2 Replies)
Discussion started by: jtelep
2 Replies

10. Shell Programming and Scripting

Convert minutes to hours, minutes, seconds

How would you convert lets say a 1000 minutes to hours, minutes, seconds (1 Reply)
Discussion started by: Vozx
1 Replies
Login or Register to Ask a Question