Extract lines if string found from last 30 min only


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
Old 02-12-2019
Extract lines if string found from last 30 min only

Hi guys,

Appreciate your help as I am stuck with searching the logs for last 30 minutes from the current time. Current time is time when you execute the script and it will search for <string> through the logs for last 30 minutes only and if <string> found then print those lines only.


The logfile has 2 different dates as shown below but searching should limit to the lines which are
  • (1) Scanning should started with syntax <Feb 12,----date----PM UTC> as shown below and
  • (2) Scanning should avoid the scanning of lines (2019-02-12T12:26:59.842+0000: 45.152Smilie
I tried various awk and sed option but unable to scan the logs for last 30min. Using grep <string>, it does the scanning for <string>, pull all lines even from previous day as per string pattern match but I want to restrict the search string and print logs for last 30 min only if the strings match exist else no data to be returned.

logfile has below entries :
Code:
<Feb 12, 2019, 12:26:54,974 PM UTC> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Feb 12, 2019, 12:26:55,687 PM UTC> <Warning> <RMI> <BEA-080099> <RMIDiagnosticUtil.startObserver scheduled diag TimerTask.>
2019-02-12T12:26:59.842+0000: 45.152: [GC [PSYoungGen: 804554K->82927K(822784K)] 906587K->210120K(2627584K), 0.1191540 secs] [Times: user=0.41 sys=0.08, real=0.12 secs]
<Feb 12, 2019, 12:27:02,40 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
--------------------------------------------------------------------------------------

# 2  
Old 02-12-2019
Please provide information on your attempts to resolve.

Until so, we will refrain from sharing any guidance.

The purpose of this Board is the assist users in solving their problems. We are not a coding service. Further, we are not a homework service - and sometimes posts appear to be an attempt to have someone solve a school assignment.

Finally, we urge all members of the Forum to NOT post a solution to this question until effort to resolve is demonstrated.
This User Gave Thanks to joeyg For This Post:
Neo (02-12-2019)
# 3  
Old 02-12-2019
Welcome to the forum.


We like and probably are able to provide help to further you from and beyond the point(s) where you're stuck. So please show us the "various awk and sed option"s you tried, and also indicate where and how they failed. Be aware that the date format of the lines you target is way more difficult to track than the one of the lines you want avoided. Does your scan need to cross midnight? Are the log entries ascending in time? Are the to-be-avoided lines interspersed regularly? By the minute?
This User Gave Thanks to RudiC For This Post:
Neo (02-12-2019)
# 4  
Old 02-12-2019
Code:
#!/bin/bash

to=`date +"<%b%_d, %Y,%l:%M:%S,%3N %p %Z>"`
let from_in_seconds=`date +%s`-5000
from=`date -d @$from_in_seconds +"<%b%_d, %Y,%l:%M:%S,%3N %p %Z>"`
awk '$0>=from && $0<=to' from="$from" to="$to" file.log

Below string matches the date format that I have it in logs but awk not working

Code:
$date +"<%b%_d, %Y,%l:%M:%S,%3N %p %Z>"
 <Feb12, 2019, 1:36:55,448 PM UTC>


below string provide the dates past 30 minutes but while using it in awk it won;t work.

Code:
date --date='30 minutes ago' -u '+%b%_d, %Y, %T,%3N %p %Z'
 Feb12, 2019, 13:13:03,306 PM UTC




Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!


--- Post updated at 01:50 PM ---

yes, <data> keep logging from the application regularly day and night in ascending order. Although, I am least concerned about the avoiding the date format (2019-02-12T12:26:59.842+0000: 45.152) but would like to have lines pulled when string matches that are from last 30 minutes only which has below date format.
<Feb 12, 2019, 12:26:54,974 PM UTC>

Last edited by RudiC; 02-12-2019 at 08:45 AM.. Reason: Added CODE tags.
# 5  
Old 02-12-2019
Hmmm - I'm a bit surprised that Feb12, 2019, 13:13:03,306 PM UTC should be considered a valid time stamp (whereas12:27:02,40 PM is). And, of course, Feb12 will never match Feb 12 in your log files.
It would be nice if your input sample would stretch across crucial points in time like midnight or 13:00h i.e. 1 PM.

Could you answer the remaining questions as well?
These 2 Users Gave Thanks to RudiC For This Post:
Neo (02-12-2019) nezabudka (02-12-2019)
# 6  
Old 02-12-2019
Code:
#!/bin/bash

NOW=$(`date +%s`)
last=$(( $NOW - 30*60 )) # last 30 minute
while read mth dy hhmmss A9 ; do

curr-time=$(date --date "${mth} ${dy} ${hhmmss}" '+%s')
if [[ "$curr-time" -ge "$last" ]] ; then
echo "${mth} ${dy} ${hhmmss} ${A9}"
fi
done < log.out



tried to use epoch (%s) but not able to use the date format which i have it in log file and +%s as it was giving the invalid date error :


Code:
-bash: curr-time=: command not found
- locked <0x000000050ef88b10> (a java.lang.Object)


Last edited by Neo; 02-12-2019 at 09:17 AM..
This User Gave Thanks to rockstar For This Post:
Neo (02-12-2019)
# 7  
Old 02-12-2019
Errors may occur with this key.
%_d
better try to change the format in the log and use %d
If the information is collected in a several days

--- Post updated at 14:28 ---

Code:
awk -v d="$(LANG=C date -d -30minutes -u +"%b %_d, %Y, %T,%3N %p %Z")" -F "<|>" '($2 > d) {print}' file

--- Post updated at 14:33 ---

some nanoseconds in the log have 2 digits?

--- Post updated at 15:03 ---

cut off nanoseconds
Code:
awk -v d="$(LANG=C date -d -30minutes -u +"%b %_d, %Y, %T")" -F "<|>" '
(gensub(/,[^,]*$/, "", 1, $2) > d)      {print}
' file

--- Post updated at 15:17 ---

may be PM and UTC need to save?
than:
Code:
date -d -30minutes -u +"%b %_d, %Y, %T %p %Z"
gensub(/(,[0-9]+ )([^,]*)$/, " \\2", 1, $2)


Last edited by nezabudka; 02-12-2019 at 01:01 PM..
These 2 Users Gave Thanks to nezabudka For This Post:
Neo (02-12-2019) RudiC (02-12-2019)
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
awk - (URGENT!) Print lines sort and move lines if match found High-T UNIX for Dummies Questions & Answers 1 02-02-2015 02:05 AM
Search String and extract few lines under the searched string ajayram_arya Shell Programming and Scripting 4 01-08-2014 05:38 PM
Extract lines with min value, using two field separators. pathunkathunk Shell Programming and Scripting 6 11-10-2013 07:55 AM
Integrate MIN and MAX in a string beca123456 UNIX for Dummies Questions & Answers 8 02-14-2013 04:37 AM
Move a block of lines to file if string found in the block. grep_me UNIX for Advanced & Expert Users 7 11-09-2012 11:29 AM
Get 20 lines above string found, and 35 below string SkySmart Shell Programming and Scripting 4 10-09-2012 10:22 AM
Search for a pattern,extract value(s) from next line, extract lines having those extracted value(s) AshwaniSharma09 Shell Programming and Scripting 7 07-24-2012 11:55 AM
grep log lines logged in 10 min Daniel Gate Shell Programming and Scripting 4 06-04-2012 06:48 AM
AWK script - extracting min and max values from selected lines grincz Shell Programming and Scripting 18 02-03-2012 06:24 PM
search and replace, when found, delete multiple lines, add new set of lines? DeuceLee Shell Programming and Scripting 3 11-23-2011 03:39 PM
Find min.max value if matching columns found using AWK vasanth.vadalur Shell Programming and Scripting 3 11-20-2011 10:19 AM
Grep a string and write a value to next line of found string angel12345 Shell Programming and Scripting 6 08-16-2011 11:07 AM
Find String in FileName and move the String to new File if not found us_pokiri Linux 1 07-20-2011 03:03 AM
Print lines after the search string until blank line is found prash184u Shell Programming and Scripting 3 08-19-2010 02:31 PM
Best approach for a 10 min extract out of several log files with timestamped records Browser_ice UNIX for Dummies Questions & Answers 3 11-15-2005 04:49 PM