Extract lines if string found from last 30 min only


Login or Register to Reply

 
Thread Tools Search this Thread
# 15  
Old 4 Weeks Ago
Quote:
Originally Posted by nezabudka
Thank you very much for the comments. All the above I be taken into account for the future.
And in the last remark. This is my carelessness and bug. The order of the expressions was violated.
Apparently I wanted to make something like that.
Code:
        if ($7 == "PM") a[1]+=12
        if (a[1] == 24) a[1] = "00"
        $5 = a[1] ":" a[2] ":" a[3]

... ... ...
Thank you for teaching, it was very informative.
Hi nezabudka,
I'm afraid the above code still doesn't work for anything that started with a[1]==12. If you start with 12 AM on a 12 hour clock you should end up with hour 00 on a 24 hour clock (the above code ends up with hour 12) and if you start with 12 PM on a 12 hour clock you should end up with hour 12 on a 24 hour clock (the above code ends up with hour 00).

If you don't like the code I suggested in post #9 or either of the suggestions I made in post #12 you could also try:
Code:
        $5 = (($7 == "PM") ? a[1] + 12 * (a[1] != 12) : (a[1] == 12) ? "00" : a[1]) ":" a[2] ":" a[3]

This User Gave Thanks to Don Cragun For This Post:
nezabudka (4 Weeks Ago)
# 16  
Old 1 Week Ago
Hello,

Thanks to everyone for their efforts. Sorry, was away for few days and didn't get the time to look at the solutions provided.

I tried all scripts from this forum but none of the script worked. All the scripts fetching the lines for entire day instead of last 30 min. My requirement is to pull the lines for last 30 min only.

Script used (for e.g.) :


Code:
awk -F "<|>| |, |," -v d="$(LANG=C date -d -30minutes -u +"%Y%m%d%T")" '
BEGIN   { split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", m2b)
        for(i = 1; i <= 12; i++)
        b2m[m2b[i]] = sprintf("%02d", i)
}
/^</    { line=$0
        if ( length($3) < 2 ) $3 = "0" $3
        split($5, a, ":" s)
        if ($5 == 24) $5 = "00"
        if ($7 == "PM") $5 = (a[1]+=12) ":" a[2] ":" a[3]
        $0 = $4 b2m[$2] $3 $5
        if ( d < $0 ) print line
}
' file_1.out

Code:
$ date   ############ current Date/time on Linux when I ran the script

Sat Mar  9 18:53:47 UTC 2019



Output from the above script:

Code:
<Mar 9, 2019, 1:02:39,176 AM UTC> <Error> <Coherence> <BEA-000000>

<Mar 9, 2019, 1:13:22,583 AM UTC> <Error> <Coherence> <BEA-000000>
<Mar 9, 2019, 1:47:08,198 AM UTC> <Error> <Coherence> <BEA-000000>
<Mar 9, 2019, 5:16:42,24 AM UTC> <Error> <Coherence> 

<Mar 9, 2019, 6:50:41,556 PM UTC> <Error> <Coherence> <BEA-000000>
<Mar 9, 2019, 6:56:45,132 PM UTC> <Error> <Coherence> <BEA-000000>


Please suggest as I need to pull last 30 min lines only whenever i execute this script not for the entire day.




Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!

Last edited by RudiC; 1 Week Ago at 02:35 PM.. Reason: Added CODE tags.
# 17  
Old 1 Week Ago
How about carefully reading, understanding, and heeding all the posts (and comments therein) offering help to you? The script you used was commented on and improved in a later post. You shouldn't expect turnkey solutions (although those are frequently delivered) but understand the proposals and experiment with them until they satisfy your needs.


Having said that, how about
Code:
$ paste -d'\t\b' <(date -f <(sed 's/^<\|>.*$//g; s/,//2' file) +"%F %T") file | awk -F"\t" -vTS="$(date -d'30 min ago' +'%F %T')" '($1 > TS) {sub ("^" $1 FS, ""); print}'
<Mar 9, 2019, 6:50:41,556 PM UTC> <Error> <Coherence> <BEA-000000>
<Mar 9, 2019, 6:56:45,132 PM UTC> <Error> <Coherence> <BEA-000000>

# 18  
Old 1 Week Ago
Hi RudiC,

Yes indeed, I have checked all posts before replying back and clearly mentioned that none of the <scripts> worked because I tried all of them. The previous post had just one of the example as don't want to bump over with all outcomes as they all produced the same outcome.

Intention here is to resolve the issue to get experts advise to get issue resolved as pulling data with dates are extremely difficult due to presence of >=2 date formats in log file.


Below script result into invalid date
Code:
$ paste -d'\t\b' <(date -f <(sed 's/^<\|>.*$//g; s/,//2' file) +"%F %T") file | awk -F"\t" -vTS="$(date -d'30 min ago' +'%F %T')" '($1 > TS) {sub ("^" $1 FS, ""); print}'

date: invalid date `2019-03-04T11:03:16.576+0000: 1392540.816: [GC [PSYoungGen: 934720K-'
date: invalid date `\tat java.lang.reflect.Method.invoke(Method.java:606)'


Last edited by Don Cragun; 6 Days Ago at 03:56 PM.. Reason: Add CODE tags again.
# 19  
Old 6 Days Ago
Quote:
Originally Posted by rockstar
Hi RudiC,

Yes indeed, I have checked all posts before replying back and clearly mentioned that none of the <scripts> worked because I tried all of them. The previous post had just one of the example as don't want to bump over with all outcomes as they all produced the same outcome.

Intention here is to resolve the issue to get experts advise to get issue resolved as pulling data with dates are extremely difficult due to presence of >=2 date formats in log file.


Below script result into invalid date
Code:
$ paste -d'\t\b' <(date -f <(sed 's/^<\|>.*$//g; s/,//2' file) +"%F %T") file | awk -F"\t" -vTS="$(date -d'30 min ago' +'%F %T')" '($1 > TS) {sub ("^" $1 FS, ""); print}'

date: invalid date `2019-03-04T11:03:16.576+0000: 1392540.816: [GC [PSYoungGen: 934720K-'
date: invalid date `\tat java.lang.reflect.Method.invoke(Method.java:606)'

Hi rockstar,
I'm very happy that you had other important matters that kept you away from this thread for a few days after you had given us your assignment to work on in your absence. I'm very sorry that we were not able to give you code that worked in your unspecified environment. I apologize for not responding on this issue for the last four days, but I've also been busy doing other things.

We are here to help you learn how to write code to meet your needs on your own. We are not here to act as your unpaid programming staff and should not be expected to write code for you while you are away doing something else. If you're unwilling to answer questions, unwilling to show us the output each of the suggested responses produced on your system, and explain what was going wrong; then there is no reason for us to waste any time trying to help you learn how to do things like this on your own.

Just saying that a script doesn't work doesn't help anyone. I can easily state that some code that you have written doesn't work, but if I don't explain how it didn't work or why it didn't work none of us learns anything useful about the problem at hand.

Like I can tell you that using:
Code:
date -d @-5000 '%Y-%m-%d %H:%M:%S'

mimicking something you showed us in post #1 in this thread is wrong. But that doesn't help you learn how to fix it. The above code has absolutely nothing to do with what the time was half an hour ago. The above code asks the system to give you a time 5000 seconds before the UNIX Epoch (i.e. 5000 seconds before midnight on the morning of January 1, 1970 at 12:00:00 AM GMT). Something like:
Code:
date -d now-1800seconds '%Y-%m-%d %H:%M:%S'

would come a lot closer to giving you a timestamp that occurred 30 minutes ago (and in a format that could be used to directly compare two timestamps as strings to see if one was earlier or later than the other until we get to the year 10000).

Your repeated refusal to use CODE tags when presenting sample input, sample output, and code segments shows us that you don't want us to see the actual format of the data you are processing and makes it impossible for us to guess at how a real solution to your problem would need to be written. (The moderators have attempted to clean up your posts, but we have obviously guessed incorrectly on some of your formatting or one or more of the suggested solutions provided would likely have met your needs.)

Above you say that having >=2 date formats is a problem??? You originally said there were exactly two date formats and that one of those formats was to be completely ignored. That made things easy. If there are other date formats you haven't told us about, it becomes very clear why none of the suggested solutions had a chance of working in your environment.

The fact that the date format you have given us to work with can't be directly compared to other dates in that format between the hours of 11:30pm on one day and 1:00am on the next day nor between 11:30am and 1:00pm on the same day is a nuisance that requires the date format in your sample data to be converted to a different format for comparisons, but I thought most, if not all, of the suggestions you had been given had tried to do that (and when they didn't, follow-up comments provided ways to get around those problems).

Last edited by RudiC; 6 Days Ago at 02:57 AM.. Reason: Epoch 1870 -> 1970
This User Gave Thanks to Don Cragun For This Post:
RudiC (6 Days Ago)
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
awk - (URGENT!) Print lines sort and move lines if match found High-T UNIX for Dummies Questions & Answers 1 02-02-2015 02:05 AM
Search String and extract few lines under the searched string ajayram_arya Shell Programming and Scripting 4 01-08-2014 05:38 PM
Extract lines with min value, using two field separators. pathunkathunk Shell Programming and Scripting 6 11-10-2013 07:55 AM
Integrate MIN and MAX in a string beca123456 UNIX for Dummies Questions & Answers 8 02-14-2013 04:37 AM
Move a block of lines to file if string found in the block. grep_me UNIX for Advanced & Expert Users 7 11-09-2012 11:29 AM
Get 20 lines above string found, and 35 below string SkySmart Shell Programming and Scripting 4 10-09-2012 10:22 AM
Search for a pattern,extract value(s) from next line, extract lines having those extracted value(s) AshwaniSharma09 Shell Programming and Scripting 7 07-24-2012 11:55 AM
grep log lines logged in 10 min Daniel Gate Shell Programming and Scripting 4 06-04-2012 06:48 AM
AWK script - extracting min and max values from selected lines grincz Shell Programming and Scripting 18 02-03-2012 06:24 PM
search and replace, when found, delete multiple lines, add new set of lines? DeuceLee Shell Programming and Scripting 3 11-23-2011 03:39 PM
Find min.max value if matching columns found using AWK vasanth.vadalur Shell Programming and Scripting 3 11-20-2011 10:19 AM
Grep a string and write a value to next line of found string angel12345 Shell Programming and Scripting 6 08-16-2011 11:07 AM
Find String in FileName and move the String to new File if not found us_pokiri Linux 1 07-20-2011 03:03 AM
Print lines after the search string until blank line is found prash184u Shell Programming and Scripting 3 08-19-2010 02:31 PM
Best approach for a 10 min extract out of several log files with timestamped records Browser_ice UNIX for Dummies Questions & Answers 3 11-15-2005 04:49 PM