Today (Saturday) We will make some minor tuning adjustments to MySQL.

You may experience 2 up to 10 seconds "glitch time" when we restart MySQL. We expect to make these adjustments around 1AM Eastern Daylight Saving Time (EDT) US.


Extract lines if string found from last 30 min only


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
Extract lines if string found from last 30 min only

Hi guys,

Appreciate your help as I am stuck with searching the logs for last 30 minutes from the current time. Current time is time when you execute the script and it will search for <string> through the logs for last 30 minutes only and if <string> found then print those lines only.


The logfile has 2 different dates as shown below but searching should limit to the lines which are
  • (1) Scanning should started with syntax <Feb 12,----date----PM UTC> as shown below and
  • (2) Scanning should avoid the scanning of lines (2019-02-12T12:26:59.842+0000: 45.152Smilie
I tried various awk and sed option but unable to scan the logs for last 30min. Using grep <string>, it does the scanning for <string>, pull all lines even from previous day as per string pattern match but I want to restrict the search string and print logs for last 30 min only if the strings match exist else no data to be returned.

logfile has below entries :
Code:
<Feb 12, 2019, 12:26:54,974 PM UTC> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Feb 12, 2019, 12:26:55,687 PM UTC> <Warning> <RMI> <BEA-080099> <RMIDiagnosticUtil.startObserver scheduled diag TimerTask.>
2019-02-12T12:26:59.842+0000: 45.152: [GC [PSYoungGen: 804554K->82927K(822784K)] 906587K->210120K(2627584K), 0.1191540 secs] [Times: user=0.41 sys=0.08, real=0.12 secs]
<Feb 12, 2019, 12:27:02,40 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
--------------------------------------------------------------------------------------

# 2  
Please provide information on your attempts to resolve.

Until so, we will refrain from sharing any guidance.

The purpose of this Board is the assist users in solving their problems. We are not a coding service. Further, we are not a homework service - and sometimes posts appear to be an attempt to have someone solve a school assignment.

Finally, we urge all members of the Forum to NOT post a solution to this question until effort to resolve is demonstrated.
This User Gave Thanks to joeyg For This Post:
# 3  
Welcome to the forum.


We like and probably are able to provide help to further you from and beyond the point(s) where you're stuck. So please show us the "various awk and sed option"s you tried, and also indicate where and how they failed. Be aware that the date format of the lines you target is way more difficult to track than the one of the lines you want avoided. Does your scan need to cross midnight? Are the log entries ascending in time? Are the to-be-avoided lines interspersed regularly? By the minute?
This User Gave Thanks to RudiC For This Post:
# 4  
Code:
#!/bin/bash

to=`date +"<%b%_d, %Y,%l:%M:%S,%3N %p %Z>"`
let from_in_seconds=`date +%s`-5000
from=`date -d @$from_in_seconds +"<%b%_d, %Y,%l:%M:%S,%3N %p %Z>"`
awk '$0>=from && $0<=to' from="$from" to="$to" file.log

Below string matches the date format that I have it in logs but awk not working

Code:
$date +"<%b%_d, %Y,%l:%M:%S,%3N %p %Z>"
 <Feb12, 2019, 1:36:55,448 PM UTC>


below string provide the dates past 30 minutes but while using it in awk it won;t work.

Code:
date --date='30 minutes ago' -u '+%b%_d, %Y, %T,%3N %p %Z'
 Feb12, 2019, 13:13:03,306 PM UTC




Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!


--- Post updated at 01:50 PM ---

yes, <data> keep logging from the application regularly day and night in ascending order. Although, I am least concerned about the avoiding the date format (2019-02-12T12:26:59.842+0000: 45.152) but would like to have lines pulled when string matches that are from last 30 minutes only which has below date format.
<Feb 12, 2019, 12:26:54,974 PM UTC>

Last edited by RudiC; 02-12-2019 at 09:45 AM.. Reason: Added CODE tags.
# 5  
Hmmm - I'm a bit surprised that Feb12, 2019, 13:13:03,306 PM UTC should be considered a valid time stamp (whereas12:27:02,40 PM is). And, of course, Feb12 will never match Feb 12 in your log files.
It would be nice if your input sample would stretch across crucial points in time like midnight or 13:00h i.e. 1 PM.

Could you answer the remaining questions as well?
These 2 Users Gave Thanks to RudiC For This Post:
# 6  
Code:
#!/bin/bash

NOW=$(`date +%s`)
last=$(( $NOW - 30*60 )) # last 30 minute
while read mth dy hhmmss A9 ; do

curr-time=$(date --date "${mth} ${dy} ${hhmmss}" '+%s')
if [[ "$curr-time" -ge "$last" ]] ; then
echo "${mth} ${dy} ${hhmmss} ${A9}"
fi
done < log.out



tried to use epoch (%s) but not able to use the date format which i have it in log file and +%s as it was giving the invalid date error :


Code:
-bash: curr-time=: command not found
- locked <0x000000050ef88b10> (a java.lang.Object)


Last edited by Neo; 02-12-2019 at 10:17 AM..
This User Gave Thanks to rockstar For This Post:
# 7  
Errors may occur with this key.
%_d
better try to change the format in the log and use %d
If the information is collected in a several days

--- Post updated at 14:28 ---

Code:
awk -v d="$(LANG=C date -d -30minutes -u +"%b %_d, %Y, %T,%3N %p %Z")" -F "<|>" '($2 > d) {print}' file

--- Post updated at 14:33 ---

some nanoseconds in the log have 2 digits?

--- Post updated at 15:03 ---

cut off nanoseconds
Code:
awk -v d="$(LANG=C date -d -30minutes -u +"%b %_d, %Y, %T")" -F "<|>" '
(gensub(/,[^,]*$/, "", 1, $2) > d)      {print}
' file

--- Post updated at 15:17 ---

may be PM and UTC need to save?
than:
Code:
date -d -30minutes -u +"%b %_d, %Y, %T %p %Z"
gensub(/(,[0-9]+ )([^,]*)$/, " \\2", 1, $2)


Last edited by nezabudka; 02-12-2019 at 02:01 PM..
These 2 Users Gave Thanks to nezabudka For This Post:
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Search String and extract few lines under the searched string

Need Assistance in shell programming... I have a huge file which has multiple stations and i wanted to search particular station and extract few lines from it and the rest is not needed Bold letters are the stations . The whole file has multiple stations . Below example i wanted to search... (4 Replies)
Discussion started by: ajayram_arya
4 Replies

2. Shell Programming and Scripting

Extract lines with min value, using two field separators.

I have a file with two ID columns followed by five columns of counts in fraction form. I'd like to print lines that have a count of at least 4 (so at least 4 in the numerator, e.g. 4/17) in at least one of the five columns. Input file: comp51820_c1_seq1 693 0/29 0/50 0/69 0/36 0/31... (6 Replies)
Discussion started by: pathunkathunk
6 Replies

3. UNIX for Dummies Questions & Answers

Integrate MIN and MAX in a string

I need to use awk for this task ! input (fields are separated by ";"): 1%2%3%4%;AA 5%6%7%8%9;AA 1%2%3%4%5%6;BB 7%8%9%10%11%12;BBIn the 1st field there are patterns composed of numbers separated by "%". The 2nd field define groups (here two different groups called "AA" and "BB"). Records... (8 Replies)
Discussion started by: beca123456
8 Replies

4. UNIX for Advanced & Expert Users

Move a block of lines to file if string found in the block.

I have a "main" file which has blocks of data for each user defined by tags BEGIN and END. BEGIN ID_NUM:24879 USER:abc123 HOW:47M CMD1:xyz1 CMD2:arp2 STATE:active PROCESS:id60 END BEGIN ID_NUM:24880 USER:def123 HOW:4M CMD1:xyz1 CMD2:xyz2 STATE:running PROCESS:id64 END (7 Replies)
Discussion started by: grep_me
7 Replies

5. Shell Programming and Scripting

Get 20 lines above string found, and 35 below string

i want to search a log for a string. when that string is found, i want to grab the a set number of lines that came before the string, and a set number of lines that come after the string. so if i search for the word "Error" in the /var/log/messages file, how can I output the 20 lines that came... (4 Replies)
Discussion started by: SkySmart
4 Replies

6. Shell Programming and Scripting

grep log lines logged in 10 min

A log files has lines (1 line per each log for a majority; a few for 2 lines per each log) May 31 14:00:11 rtprodapp1 local2:notice sudo: jdoe : TTY=pts/0 ; PWD=/home/jdoe ; USER=root ; COMMAND=/usr/bin/su - May 31 14:03:19 rtprodapp1 local2:notice sudo: jdoe : TTY=pts/0 ; PWD=/home/jdoe ;... (4 Replies)
Discussion started by: Daniel Gate
4 Replies

7. Shell Programming and Scripting

search and replace, when found, delete multiple lines, add new set of lines?

hey guys, I tried searching but most 'search and replace' questions are related to one liners. Say I have a file to be replaced that has the following: $ cat testing.txt TESTING AAA BBB CCC DDD EEE FFF GGG HHH ENDTESTING This is the input file: (3 Replies)
Discussion started by: DeuceLee
3 Replies

8. Shell Programming and Scripting

Find min.max value if matching columns found using AWK

Input_ File : 2 3 4 5 1 1 0 1 2 1 -1 1 2 1 3 1 3 1 4 1 6 5 6 6 6 6 6 7 6 7 6 8 5 8 6 7 Desired output : 2 3 4 5 -1 1 4 1 6 5 6 8 5 8 6 7 (3 Replies)
Discussion started by: vasanth.vadalur
3 Replies

9. Shell Programming and Scripting

Print lines after the search string until blank line is found

All I want is to look for the pattern in the file...If I found it at # places... I want print lines after those pattern(line) until I find a blank line. Log EXAMPLE : MT:Exception caught The following Numbers were affected: 1234 2345 2346 Error java.lang.InternalError:... (3 Replies)
Discussion started by: prash184u
3 Replies

10. UNIX for Dummies Questions & Answers

Best approach for a 10 min extract out of several log files with timestamped records

I have a task where I need to code a shell script to extract a 10 min range (10 min from now until now) extract of a log file. I taught I could simply use a command that would say something like Start=date - 10 min but I didn't find anything. Looks like the only solution would have to code a... (3 Replies)
Discussion started by: Browser_ice
3 Replies

Featured Tech Videos