Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Search for text between two time frame using sed

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Search for text between two time frame using sed
# 1  
Old 02-02-2013
Question Search for text between two time frame using sed
I have log files with time stamps. I want to search for text between two time stamp using sed even if the first tme stamp or the last time stamp are not present. For e.g. if i search between 9:30 and 9:40 then it should return text even if 9:30 or 9:40 is not there but between 9:30 and 9:40 is present.
I am using a sed one liner:
Code:
sed -n '/7:30:/,/7:35:/p' xyz.log

But it only returns data if both the time stamps are present, it will print everything if one of the time stamp are missing. An if the time is in 12 hr format it will pull data for both AM and PM.
Additionally , i have different time stamps formats for different log files so i need a generic command.
following are some time format examples e.g.
Code:
<Jan 27, 2013 12:57:16 AM MST>Jan 29, 2013 8:58:12 AM 2013-01-31 06:44:04,883

One of them contains AM/PM i.e. 12 hr format and other cntains 24 hr format so i have to account for that as well.
I have tried this as weel but it doesnt work:
Code:
sed -n -e '/^2012-07-19 18:22:48/,/2012-07-23 22:39:52/p' history.log

Please help
Last edited by Scott; 02-02-2013 at 10:02 PM.. Reason: Please use code tags
# 2  
Old 02-02-2013
Quote:
Originally Posted by Azher
Additionally , i have different time stamps formats for different log files so i need a generic command.
Can you post few sample formats including text that you want to extract and desired output?
# 3  
Old 02-02-2013
Some of the time formats

--> <Jan 27, 2013 12:57:16 AM MST>
--> Jan 29, 2013 8:58:12 AM
--> 2013-01-31 06:44:04,883

Entry can be anything like:

Code:
 
--> 2013-01-31 06:44:04,883 Error in the log file etc...
--> <Jan 27, 2013 12:57:16 AM MST> Logger initialized etc...

Now i want to get everything between 7:30 and 7:40

Code:
 
--> 2013-01-28 07:33:57,903 [Main Thread] = INFO (CugRefresher.java:75) - 
-->2013-01-28 07:34:00,788 [Main Thread] = INFO (CugRefresher.java:154) 
-->2013-01-28 07:35:01,765 [Main Thread] = INFO (CugRefresher.java:164) 
-->2013-01-28 07:40:00,780 [Main Thread] = INFO (CugRefresher.java:194)

So i want to get data between 7:30 and 7:37 so it should return:

Code:
 
-->2013-01-28 07:34:00,788 [Main Thread] = INFO (CugRefresher.java:154) 
-->2013-01-28 07:35:01,765 [Main Thread] = INFO (CugRefresher.java:164)

# 4  
Old 02-02-2013
I don't think what you are trying to do is possible using sed!

You have to perform date arithmetic here and fetch data between the specified range.

Here is a BASH script that might help you:
Code:
#!/bin/bash

date_1=$1
date_2=$2

e_date_1=$( date -d"$date_1" +"%s" )
e_date_2=$( date -d"$date_2" +"%s" )

while IFS=, read dt skip
do
        e_dt=$( date -d"$dt" +"%s" )
        if [ $e_dt -ge $e_date_1 ] && [ $e_dt -le $e_date_2 ]
        then
                printf "${dt},${skip}\n"
        fi
done < infile

Here is what I have in file: infile
Code:
$ cat infile
2013-01-28 07:33:57,903 [Main Thread] = INFO (CugRefresher.java:75)
2013-01-28 07:34:00,788 [Main Thread] = INFO (CugRefresher.java:154)
2013-01-28 07:35:01,765 [Main Thread] = INFO (CugRefresher.java:164)
2013-01-28 07:40:00,780 [Main Thread] = INFO (CugRefresher.java:194)

Here is the script output:
Code:
$ ./extract_bw_dates.sh "2013-01-28 07:30:00" "2013-01-28 07:37:00"
2013-01-28 07:33:57,903 [Main Thread] = INFO (CugRefresher.java:75)
2013-01-28 07:34:00,788 [Main Thread] = INFO (CugRefresher.java:154)
2013-01-28 07:35:01,765 [Main Thread] = INFO (CugRefresher.java:164)

I hope this helps.
# 5  
Old 02-03-2013
This is simple to do with awk
Code:
awk '$0>=from && $0<=to' from="2013-01-28 07:30" to="2013-01-28 07:37" infile
or
awk '$0>="2013-01-28 07:30" && $0<="2013-01-28 07:37"' infile

Code:
2013-01-28 07:33:57,903 [Main Thread] = INFO (CugRefresher.java:75)
2013-01-28 07:34:00,788 [Main Thread] = INFO (CugRefresher.java:154)
2013-01-28 07:35:01,765 [Main Thread] = INFO (CugRefresher.java:164)

---------- Post updated at 09:34 ---------- Previous update was at 09:25 ----------

Here is an example to get last 4 hour of your log.
Code:
awk '$0>=from && $0<=to' from="$(date +%Y"-"%m"-"%d" "%H:%M -d -4hour)" to="$(date +%Y"-"%m"-"%d" "%H:%M)" infile

# 6  
Old 02-03-2013
@Jotne How does this awk command will take care of the AM/PM in the timestamp for 12hr format?

---------- Post updated at 05:25 AM ---------- Previous update was at 04:10 AM ----------

The above awk solution wont work as i have to deal with different time formats as earlier mentioned and it the awk solution requires a fix time format

Like i said my time format can be anyhting like shown below:

Code:
 
<Jan 27, 2013 12:57:16 AM MST>
Jan 29, 2013 8:58:12 AM
2013-01-31 06:44:04,883

Pleae help...I am trying for something like AWK or SED or any regex combination
# 7  
Old 02-04-2013
I think you need to convert your log to 24 hour format before using the command.
Since I live in an area without AM/PM, I do not need to deal with this type of problem.
Here is some solution I use to convert log automatically
Code:
tail --follow=name /var/log/syslog | awk '!/found|cache/ {print | "tee /var/log/new.log"}' &

This remove all log entry with found & cache, buy you can convert to 24 hour format.
Then do all test on new.log
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Need to filter the result set within 2 time frame

my sample file is like this $cat onefile 05/21/18 13:10:07 ABRT US1CPDAY Status 1 05/21/18 21:18:54 ABRT DailyBackup_VFFPRDAPENTL01 Status 6 05/21/18 21:26:24 ABRT DailyBackup_VFFPRDAPENTL02 Status 6 05/21/18 21:57:36 ABRT DailyBackup_vm-ea1ffpreng01 Status 6... (7 Replies)
Discussion started by: gotamp
7 Replies

2. Shell Programming and Scripting

Help on script to capture info on log file for a particular time frame

Hi I have a system running uname -a Linux cmovel-db01 2.6.32-38-server #83-Ubuntu SMP Wed Jan 4 11:26:59 UTC 2012 x86_64 GNU/Linux I would like to capture the contents of /var/log/syslog from 11:00AM to 11:30AM and sent to this info via email. I was thinking in set a cron entry at that... (2 Replies)
Discussion started by: fretagi
2 Replies

3. UNIX for Dummies Questions & Answers

UNIX Account getting Locked Everyday between same Time Frame

I am facing an Issue with a particular Unix Account ( ie a particular Userid) getting LOCKED everyday between 7:30am and 8:00am. The Password associated with this particular Account has been setup such that it should never Expire at all but it does LOCK the Account after more than 3 failed... (5 Replies)
Discussion started by: pchegoor
5 Replies

4. Shell Programming and Scripting

awk : collecting all data between two time frame

Hi Experts , I need your help to collect the complete data between two time frame from the log files, when I try awk it's collecting the data only which is printed with time stamp for example, awk works well from "16:00 to 17:30" but its not collecting <line*> "from 17:30 to 18:00" ... (8 Replies)
Discussion started by: zenkarthi
8 Replies

5. Shell Programming and Scripting

awk : Search for text between two time frame (12 hours)

I have created the script to grep the errors from weblogic logs files and redirecting output to file.txt ...From file.txt I'm using awk command to collect the past 20 mins output...The script running from cron every 15 mins... The script working well... Now the challenges, I'm trying to use... (27 Replies)
Discussion started by: zenkarthi
27 Replies

6. Shell Programming and Scripting

Shell Script to delete files within a particular time frame under multiple sub folders

Greetings! I'm looking for starting information for a shell script. Here's my scenario: I have multiple folders(100) for example: /www/test/applications/app1/logs /www/test/applications/app2/logs Within these folders there are log files files that need to be deleted after a month. ... (3 Replies)
Discussion started by: whysolucky
3 Replies

7. Shell Programming and Scripting

Search text and append using SED?

I have file . cat hello.txt Hello World I would like to append a string "Today " so the output is cat hello.txt Hello World Today I dont know which line number does the "Hello World" appears otherwise I could have used the Line number to search and append . (3 Replies)
Discussion started by: gubbu
3 Replies

8. Shell Programming and Scripting

search string during a specific time frame

Can someone please help me with searching a string during a specific time frame. Below is the format of the time from my log file. "GET /AAM2009_wherewereheaded.wmv HTTP/1.1" 200 52307085 The search string I need is "AAM2009_wherewereheaded.wmv" I need to search the number of... (1 Reply)
Discussion started by: tadi18
1 Replies

9. Shell Programming and Scripting

Sed Help.To Search Between Pattern1 And Pattern2 Containing Certain Text

Hi, Here is a sample of my Test File $ cat TestFile1 Prompt Table DQZ_ALTER_SCHEMA_ID; ALTER TABLE DQZ.DQZ_ALTER_SCHEMA_ID MONITORING; ALTER TABLE DQZ.DQZ_ALTER_SCHEMA_ID STORAGE ( NEXT 3464K ); Prompt Table DQZ_ALTER_SCHEMA_ID; ALTER TABLE DQZ.DQZ_ALTER_SCHEMA_ID MOVE LOB... (16 Replies)
Discussion started by: rajan_san
16 Replies
Login or Register to Ask a Question