Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Collecting logs between two time stamps

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Collecting logs between two time stamps
# 1  
Old 03-29-2015
Collecting logs between two time stamps
Hi, please help me to collect the entire log files between two time stamp.

for example,
I am looking script to collect the entire log between "2015-03-27 15:59" to "2015-03-27 16:15" in the below sample log file.


OS : RHEL 6.3
Date/Time : 24 hours format, the time is printing each log file but the date won't print all the log entry. Also time printing starting and middle (after the date)



Sample logs:

Code:
15:59:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 15:59:07,091:INFO   server design Pool -- 1066: bundles initialized                                 [system]: Console 
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-13) Creating Service {http://xyz.com/sche/claas/Acco
15:59:08,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-18) Creating Service {http://xyz.com/sche/wor/tt
15:59:0,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1800) Creating Service {http://xyz.com/sche/claas/Add
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-2000) Creating Service {http://xyz.com/sche/cldaas/Acco
15:59,07 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-222) Creating Service {http://xyz.com/egg/asdd/ssco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-133) Creating Service {http://xyz.com/sche/claas/Acco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1344) Creating Service {http://xyz.com/sche/claas/Acco
16:05:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 16:05:07,091:INFO   server Thread Pool -- 86: bundles initialized                                 [system]: Console
16:05:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-13) Creating Service {http://xyz.com/sche/claas/Acco
16:05:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-18) Creating Service {http://xyz.com/sche/wor/tt
16:06:010,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 16:06:10,091:INFO   server Thread Pool -- 86: bundles initialized                                 [system]: Console
aaaaa
bbbbb
cccccc
16:15:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 16:15:07,091:INFO   server design Pool -- 106: bundles initialized

:

Appreciate your help.
# 2  
Old 03-29-2015
Most log files have dates and times in monotonically non-decreasing order. Yours does not.
Code:
15:59:07
15:59:08
15:59:0
15:59:08
15:59:07
16:05:07
16:06:010
16:15:07

Most log files have timestamps in a consistent format. Yours does not.
Code:
15:59:0
15:59:08
16:06:010

Assuming that comma and space are field delimiters, is the date field on lines that contain dates always field 9?

Since timestamps aren't in order, how are we supposed to guess what date should apply to a line that does not have a date?

What is supposed to happen to log file lines that do not have a date or timestamp?

What output are you hoping to produce for the sample input you provided?
# 3  
Old 03-29-2015
Don, Thanks for the reply.

Yes, the date is printing the same filed. I'm excepting the below output if I grep the time stamp from "2015-03-27 15:59" to "2015-03-27 16:05"


Code:
15:59:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 15:59:07,091:INFO   server design Pool -- 1066: bundles initialized                                 [system]: Console 
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-13) Creating Service {http://xyz.com/sche/claas/Acco
15:59:08,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-18) Creating Service {http://xyz.com/sche/wor/tt
15:59:0,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1800) Creating Service {http://xyz.com/sche/claas/Add
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-2000) Creating Service {http://xyz.com/sche/cldaas/Acco
15:59,07 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-222) Creating Service {http://xyz.com/egg/asdd/ssco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-133) Creating Service {http://xyz.com/sche/claas/Acco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1344) Creating Service {http://xyz.com/sche/claas/Acco
16:05:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 16:05:07,091:INFO   server Thread Pool -- 86: bundles initialized

# 4  
Old 03-29-2015
Thank you for responding.
In my previous post, I asked four questions. You answered one of them.
If you choose not to answer the other three questions I asked in my previous post, I can't determine how to guess correctly at what you should be done.

And, for the record, greping won't work for what you're asking for. The grep utility uses regular expressions to look for certain patterns. It is not suitable for performing alphanumeric range comparisons.
# 5  
Old 03-30-2015
Apologies, here is my reply for your questions...

Assuming that comma and space are field delimiters, is the date field on lines that contain dates always field 9?

Ans: The date's are printing always the same field.Sorry, is that 9 or 10?

Code:
12:18:17,734 INFO  [xyzout] (ajp-/xx.xx.xx.xxx:1234-1) 2015-03-30 12:18:17,733:INFO   :xyz-/xx.xx.xx.xxx:1234-1: smopBindingStub object in ABCDDataRetrievalVOg class
      [name_nname@email.domain.com.smb]: workingCreateQuotesFromabcdefData
12:20:17,734 INFO  [xyzout] (ajp-/xx.xx.xx.xxx:1234-21) 2015-03-30 12:20:17,733:INFO   :xyz-/xx.xx.xx.xxx:1234-1: smopBindingStub object in ABCDDataRetrievalVOg class
      [name_nname@email.domain.com.smb]: workingCreateQuotesFromabcdefData
12:25:17,734 INFO  [xyzout] (ajp-/xx.xx.xx.xxx:1234-21) 2015-03-30 12:25:17,733:INFO   :xyz-/xx.xx.xx.xxx:1234-1: smopBindingStub object in ABCDDataRetrievalVOg class
      [name_nname@email.domain.com.smb]: workingCreateQuotesFromabcdefData
12:26:17,734 INFO  [xyzout] (ajp-/xx.xx.xx.xxx:1234-22) 2015-03-30 12:26:17,733:INFO   :xyz-/xx.xx.xx.xxx:1234-1: smopBindingStub object in ABCDDataRetrievalVOg class
      [name_nname@email.domain.com.smb]: workingCreateQuotesFromabcdefData


Since timestamps aren't in order, how are we supposed to guess what date should apply to a line that does not have a date?

Ans: It seems like, the data is printing the first paragraph and time alone is printing the remaining lines so I can get all middle lines which doesn't have date if I collect the logs between two date and time.


What is supposed to happen to log file lines that do not have a date or timestamp?

Ans: Those are printing between two different date (or) timestamp. If that possible to collect all the logs lines between two different date or timestamp? For example, past 20 mins logs.

What output are you hoping to produce for the sample input you provided?

Ans:
Code:
15:59:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 15:59:07,091:INFO   server design Pool -- 1066: bundles initialized                                 [system]: Console 
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-13) Creating Service {http://xyz.com/sche/claas/Acco
15:59:08,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-18) Creating Service {http://xyz.com/sche/wor/tt
15:59:0,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1800) Creating Service {http://xyz.com/sche/claas/Add
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-2000) Creating Service {http://xyz.com/sche/cldaas/Acco
15:59,07 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-222) Creating Service {http://xyz.com/egg/asdd/ssco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-133) Creating Service {http://xyz.com/sche/claas/Acco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1344) Creating Service {http://xyz.com/sche/claas/Acco
16:05:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 16:05:07,091:INFO   server Thread Pool -- 86: bundles initialized

Also, can you please explain me the below script.

Code:
awk -F, '
function dform(v){
                    gsub(/[-:]/," ",v)
                    return mktime(v)
                 }
           NR==1 {
                    start = dform(start)
                     end  = dform(end)
                 }
    dform($1)!=-1{
                    f = dform($1)>=start && dform($1) <= end ? 1 : 0
                 }f
        ' start="$from" end="$to" $log1 > /script/xyz.txt

# 6  
Old 03-30-2015
What has this
Code:
12:18:17,734 INFO  [xyzout] (ajp-/xx.xx.xx.xxx:1234-1) 2015-03-30 12:18:17,733:INFO . . . 
      [name_nname@email.domain.com.smb]: workingCreateQuotesFromabcdefData

to do with your problem resp. the sample given?

Code:
awk -F, '                                                               # run awk with field separator set to ","
function dform(v){                                                      # convert time string to binary representation
                    gsub(/[-:]/," ",v)                                  # replace "-" and ":" with " " in v
                    return mktime(v)                                    # do conversion
                 }
           NR==1 {                                                      # at start of file
                    start = dform(start)                                # convert start parameter
                     end  = dform(end)                                  # and end
                 }
    dform($1)!=-1{                                                      # if first field is a valid time string
                    f = dform($1)>=start && dform($1) <= end ? 1 : 0    # set "doprint" var "f" if $1 within boundaries   
                 }f                                                     # print line if "f"
        ' start="$from" end="$to" $log1 > /script/xyz.txt               # supply boundaries and log file; redirect

Be aware that mktime is not available in all awk versions.
You could use this snippet for your problem if you use [ ,] for the field separator and apply it to $10 instead of $1. It would not print the last line in your desired output, though.
# 7  
Old 03-30-2015
Thank you...

Yes, I gave him sample output to understand the date field.

please correct me, the below script not helping to get my excepted output...

Code:
#log
log1=/app/applicationdir/log/test.log

# Time stamp
to=`date "+%Y-%m-%d %H:%M:%S"`
from=`date "+%Y-%m-%d %H:%M:%S" -d "-20 minutes"`

#echo $from

awk -F, '
function dform(v){
                    gsub(/[-:]/," ",v)
                    return mktime(v)
                 }
           NR==1 {
                    start = dform(start)
                     end  = dform(end)
                 }
    dform($10)!=-1{
                    f = dform($10)>=start && dform($10) <= end ? 1 : 0
                 }f
        ' start="$from" end="$to" $log1 > /tmp/xyzlog.txt
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Syslog-ng not working not collecting logs on rhel

Hi, I need help on syslog-ng on RHEL 7.2. It is working as expected. As per configuration, it is supposed to create authlog, messages and xymessages daily in respective folder of date. But I can see only messages file and that is also not updating well. # ps -ef | grep -i syslog root 22954... (1 Reply)
Discussion started by: solaris_1977
1 Replies

2. Windows & DOS: Issues & Discussions

Cygwin_openssh time stamps

I've installed cygwin_openssh on Windows 2012 R2 and it's working great. My issue is when a file is uploaded say from a different timezone, when it is uploaded, it doesnt pick up the sftp servers time.. Is there a way to fix that? i.e. When someone in PST uploads a file to this server in EST,... (0 Replies)
Discussion started by: MikeAdkins
0 Replies

3. Shell Programming and Scripting

How to get the Logs between two Time Stamps?

Hi, I have been working on the error Log script, where errors are pulled from server. I need to pull the data of the error logs between two dates & time, for example : 22/12/2014 20:00:00 22/12/2014 22:00:00 Whatever error have came during this duration. Now the question is the record... (6 Replies)
Discussion started by: amitgpta90
6 Replies

4. Shell Programming and Scripting

Increment time stamps.

Hi Gents. Please can you help me to solve a problem. I have a long list of files, which I need to change the time stamp. -r--r--r-- 1 geo2 geovect 47096216 Feb 8 10:40 00000009.segd -r--r--r-- 1 geo2 geovect 47096216 Feb 8 10:40 00000010.segd -r--r--r-- 1 geo2 geovect 47096216 Feb ... (11 Replies)
Discussion started by: jiam912
11 Replies

5. Shell Programming and Scripting

date time stamps in bash

I'm looking for a way to have the "date" command output the date in a specific format. I'm not familiar with the different ways to use the date command at all. i read up on it, but i dont get how to manipulate it. i know that i can get the date format to give me a format like: 2012-10-13... (6 Replies)
Discussion started by: SkySmart
6 Replies

6. Shell Programming and Scripting

Time difference between two time stamps

Hi Friends, I have 2 varaibles which contain START=`date '+ %m/%d/%y %H:%M:%S'` END=`date '+ %m/%d/%y %H:%M:%S'` i want the time difference between the two variables in Seconds. Plz help. (2 Replies)
Discussion started by: i150371485
2 Replies

7. Red Hat

Collecting logs by running command

Hi, i am a general user of linux but we work mostly on windows next i am moving full time on linux. here is my question: We have product which consist or several subsystem each subsystem has one module to create logs file dump. and i am going to write that support dump tool. we need to... (3 Replies)
Discussion started by: ajayyadavmca
3 Replies

8. Shell Programming and Scripting

comparing time stamps

Hello All, I'm generating timestamps (file creation timestamps) for all the files in a directory. I need to compare all the timestamps. for example if i have 4 files and their timestamps are 20091125114556, 20091125114556,20091125114556,20091125114556 respectively. I need to differentiate... (9 Replies)
Discussion started by: RSC1985
9 Replies

9. UNIX for Dummies Questions & Answers

comparing time stamps

Hello All, I'm generating timestamps (file creation timestamps) for all the files in a directory. I need to compare all the timestamps. for example if i have 4 files and their timestamps are 20091125114556, 20091125114556,20091125114556,20091125114556 respectively. I need to differentiate... (1 Reply)
Discussion started by: RSC1985
1 Replies
Login or Register to Ask a Question