Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Collecting logs between two time stamps

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Collecting logs between two time stamps
# 8  
Old 03-30-2015
It can't, as you
- didn't use [ ,] as the field separator (add "*" to allow for multiple spaces between fields)
- your to and from variables are date/time format, and $10 is time only. Use $9" "$10 in lieu of $10 alone.
# 9  
Old 03-31-2015
Apologies , I am new to awk dform , Would be great if you modify the specific line or script and update here since I am getting error after changing the
Code:
 ($9,$10)

, Also not sure which line I need to replace for
Code:
[ ,]

, add "*" . Thanks!
# 10  
Old 03-31-2015
Facebook
I've done several sort of optimizing steps in one here. Try
Code:
awk -F"[ ,]*" '
function dform(v)       {gsub(/[-:]/," ",v)
                         return mktime(v)
                        }

NR==1                   {start = dform(start)
                         end   = dform(end)  
                        }

TMP=dform($9" "$10)!=-1 {f = (TMP>=start && TMP <= end)
                        }

f
 
' start="$from" end="$to"  $log1 > /tmp/xyzlog.txt

# 11  
Old 03-31-2015
It's not working as excepted...

#more test.log
Code:
14:15:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:15:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine
14:18:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:18:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine
14:28:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:28:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine
14:28:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:28:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine
14:28:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:28:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine
14:28:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:28:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine

#sh -x test.sh or
#./test.sh

Code:
+ log1=/tmp/test.log
++ date '+%Y-%m-%d %H:%M:%S'
+ to='2015-03-31 14:30:34'
++ date '+%Y-%m-%d %H:%M:%S' -d '-30 minutes'
+ from='2015-03-31 14:00:34'
+ awk '-F[ ,]*' '
function dform(v)       {gsub(/[-:]/," ",v)
                         return mktime(v)
                        }

NR==1                   {start = dform(start)
                         end   = dform(end)
                        }

TMP=dform($9" "$10)!=-1 {f = (TMP>=start && TMP <= end)
                        }

f
' 'start=2015-03-31 14:00:34' 'end=2015-03-31 14:30:34' /tmp/test.log

#more /tmp/xyzlog.txt
empty.
# 12  
Old 03-31-2015
Kidding? That input file is different in structure than the one post#1 and the output in post#3. With that input you can't expect an output close to what you desired.
And - No redirection - no output file, of course.
# 13  
Old 03-31-2015
Apologies, I am messing here...

The post#1 and post#3 and post#11, all the outputs are taken from the same logs.

Just noticed, the logs date printing in different field in each samples... I think I have to work with development team to standardize the logs pattern.

Sample -1
Code:
12:00:45,026 INFO  [stdout] (pool-92-thread-5) 2015-03-31 12:00:45,026:ERROR  :pool-92-thread-1: [1420043245026] ContextTracking [system]: YCPContext
12:00:45,027 INFO  [stdout] (pool-92-thread-5) 2015-03-31 12:00:45,026:ERRORDTL:pool-92-thread-1: [1420003245526]java.lang.RuntimeException: ContextTracking

Sample - 2
Code:
14:15:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:15:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine
14:18:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:18:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine

Sample - 3

Code:
5:59:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 15:59:07,091:INFO   server design Pool -- 1066: bundles initialized                                 [system]: Console 
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-13) Creating Service {http://xyz.com/sche/claas/Acco
15:59:08,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-18) Creating Service {http://xyz.com/sche/wor/tt
15:59:0,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1800) Creating Service {http://xyz.com/sche/claas/Add
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-2000) Creating Service {http://xyz.com/sche/cldaas/Acco
15:59,07 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-222) Creating Service {http://xyz.com/egg/asdd/ssco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-133) Creating Service {http://xyz.com/sche/claas/Acco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1344) Creating Service {http://xyz.com/sche/claas/Acco
16:05:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 16:05:07,091:INFO   server Thread Pool -- 86: bundles initialized

since I executed sh-x it's not showing the redirected file name, when I execute ksh -x or ./test.sh it showing and showing the update time stamp in the xyzlog.txt

Thanks for the help!
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Syslog-ng not working not collecting logs on rhel

Hi, I need help on syslog-ng on RHEL 7.2. It is working as expected. As per configuration, it is supposed to create authlog, messages and xymessages daily in respective folder of date. But I can see only messages file and that is also not updating well. # ps -ef | grep -i syslog root 22954... (1 Reply)
Discussion started by: solaris_1977
1 Replies

2. Windows & DOS: Issues & Discussions

Cygwin_openssh time stamps

I've installed cygwin_openssh on Windows 2012 R2 and it's working great. My issue is when a file is uploaded say from a different timezone, when it is uploaded, it doesnt pick up the sftp servers time.. Is there a way to fix that? i.e. When someone in PST uploads a file to this server in EST,... (0 Replies)
Discussion started by: MikeAdkins
0 Replies

3. Shell Programming and Scripting

How to get the Logs between two Time Stamps?

Hi, I have been working on the error Log script, where errors are pulled from server. I need to pull the data of the error logs between two dates & time, for example : 22/12/2014 20:00:00 22/12/2014 22:00:00 Whatever error have came during this duration. Now the question is the record... (6 Replies)
Discussion started by: amitgpta90
6 Replies

4. Shell Programming and Scripting

Increment time stamps.

Hi Gents. Please can you help me to solve a problem. I have a long list of files, which I need to change the time stamp. -r--r--r-- 1 geo2 geovect 47096216 Feb 8 10:40 00000009.segd -r--r--r-- 1 geo2 geovect 47096216 Feb 8 10:40 00000010.segd -r--r--r-- 1 geo2 geovect 47096216 Feb ... (11 Replies)
Discussion started by: jiam912
11 Replies

5. Shell Programming and Scripting

date time stamps in bash

I'm looking for a way to have the "date" command output the date in a specific format. I'm not familiar with the different ways to use the date command at all. i read up on it, but i dont get how to manipulate it. i know that i can get the date format to give me a format like: 2012-10-13... (6 Replies)
Discussion started by: SkySmart
6 Replies

6. Shell Programming and Scripting

Time difference between two time stamps

Hi Friends, I have 2 varaibles which contain START=`date '+ %m/%d/%y %H:%M:%S'` END=`date '+ %m/%d/%y %H:%M:%S'` i want the time difference between the two variables in Seconds. Plz help. (2 Replies)
Discussion started by: i150371485
2 Replies

7. Red Hat

Collecting logs by running command

Hi, i am a general user of linux but we work mostly on windows next i am moving full time on linux. here is my question: We have product which consist or several subsystem each subsystem has one module to create logs file dump. and i am going to write that support dump tool. we need to... (3 Replies)
Discussion started by: ajayyadavmca
3 Replies

8. Shell Programming and Scripting

comparing time stamps

Hello All, I'm generating timestamps (file creation timestamps) for all the files in a directory. I need to compare all the timestamps. for example if i have 4 files and their timestamps are 20091125114556, 20091125114556,20091125114556,20091125114556 respectively. I need to differentiate... (9 Replies)
Discussion started by: RSC1985
9 Replies

9. UNIX for Dummies Questions & Answers

comparing time stamps

Hello All, I'm generating timestamps (file creation timestamps) for all the files in a directory. I need to compare all the timestamps. for example if i have 4 files and their timestamps are 20091125114556, 20091125114556,20091125114556,20091125114556 respectively. I need to differentiate... (1 Reply)
Discussion started by: RSC1985
1 Replies
Login or Register to Ask a Question