Calculating time windows from logfile timestamps


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Calculating time windows from logfile timestamps
# 1  
Old 07-11-2012
Calculating time windows from logfile timestamps

Hey all. I am working on some scripts in bash to perform a variety of functions; there are a variety of steps involved, and they must happen in a specific sequence; what I need help with is a way to calculate some differences in a timestamp in a logfile.
One of the steps in the scripts I am writing involves issuing a command to an application that executes a 'deployment' process of sorts; the shell interface to this application basically receives the request to start this deployment process, and exits. The deployment process can take a wildly variable amount of time (a few minutes, up to a few hours); there are additional actions that my script needs to perform once that process is complete, but these actions cannot begin until it has.

The application that is performing this deployment process writes to a logfile, and I know the entry in the logfile that indicates that this deployment process has finished; however, this logfile is written to consistently, and I cannot clear it. What I need to do is to identify, within my script that initiates the deployment process, the time that the command is executed, and then search through the application's logfile for the completion condition, and compare the timestamps of those messages for the most recent one to occur after the time noted by my script when the action was begun. I can do most of this already; where I'm getting stuck is in parsing the timestamps into a useful, computable format. I can control the way in which my script sets its initial timestamp, but I cannot control the format in which the logfile marks its timestamps, which are written thusly (this entry is the completion condition that I am looking for):

Code:
Jul 10, 2012 7:47:45 PM] Application deployment complete.

The following date command will produce a timestamp formatted in exactly this fashion, but I don't know if that's actually useful or not:
Code:
date +%b\ %-d\,\ %Y\ %-l\:%M\:%S\ %p

I can certainly run the timestamp through a set of sed steps to parse out the individual pieces of information in the logfile, but I'm afraid that what I have thus far is, in addition being obviously cumbersome and probably quite amateurish, potentially unproductive and not really the right way to go about this:

Code:
#!/bin/bash
# Sets startDate variable with current timestamp in logfile's format, removes unnecessary characters, and converts to underscore delimited format
export startDate=`date +%b\ %-d\,\ %Y\ %-l\:%M\:%S\ %p | sed 's/ /_/g' | sed 's/,//g' | sed 's/:/_/g'`

# Sets each field in timestamp to individual variables using cut
export startMonth=`echo $startDate | cut -d\_ -f1`
export startDay=`echo $startDate | cut -d\_ -f2`
export startYear=`echo $startDate | cut -d\_ -f3`
export startAMPM=`echo $startDate | cut -d\_ -f7`
export startHour=`echo $startDate | cut -d\_ -f4`
# Converts 12-hour time to 24-hour time
if [ "$startAMPM" == PM];
    then export startHour=$(($startHour + 12))
fi
export startMin=`echo $startDate | cut -d\_ -f5`
export startSec=`echo $startDate | cut -d\_ -f6`

If I echo each of these variables individually at the end of the script, what I get when running it is this:
Code:
Jul 11 2012 19 53 22

I can do all this same logic on the entry in the application's logfile as well (replacing the date command with for instance, :
Code:
cat /path/to/logfile | grep "Application deployment complete"

-- but I guess my big question here is, then what?

How can I actually use that information to look for the right entry?
Also, is there a better way to parse out that information?

Thanks very much everyone, I appreciate the help.

Last edited by methyl; 07-11-2012 at 09:44 PM.. Reason: please use code tags. attempt to format post from streamed data to readability
# 2  
Old 07-11-2012
Please post what Oparating System and version you are running and what Shell you prefer.

For anything to do with date arithmetic, please post whether you have the GNU date command and/or a modern version of perl.
Please also post sample data for a date from last week (i.e. with a single digit day).
# 3  
Old 07-12-2012
Can you take note of how many lines are in the logfile when the process starts and then look for the "Application deployment complete" message on a line greater than this?
# 4  
Old 07-12-2012
Quote:
Originally Posted by methyl
Please post what Oparating System and version you are running and what Shell you prefer.

For anything to do with date arithmetic, please post whether you have the GNU date command and/or a modern version of perl.
Please also post sample data for a date from last week (i.e. with a single digit day).
Hi Methyl, thanks very much. Operating system is Red Hat Enterprise Linux Server release 5.5; I prefer the bash shell.

I do appear to have the GNU date command: the bottom of the date manpage shows it as 'date 5.97', and is dated February 2010.

A sample the log file in question:

cat application-logfile.log | grep "Application deployment complete."
[Jun 20, 2012 3:57:57 PM]: Application deployment complete.
[Jun 27, 2012 12:28:14 PM]: Application deployment complete.
[Jun 28, 2012 10:38:10 PM]: Application deployment complete.
[Jul 6, 2012 7:35:38 PM]: Application deployment complete.
[Jul 10, 2012 7:47:45 PM]: Application deployment complete.

(Single-digit days are not zero-padded, which is why I used %-d in my test date command).

Regarding Perl, if it is included by default in the RHEL 5.5 Server release, then I probably do; I really don't know anything at all about Perl, and sadly have tended to avoid it accordingly. I'm sure it's much easier to do all this with a Perl script, but that's a whole other can of worms I'm hesitant to open. =\

---------- Post updated at 11:07 PM ---------- Previous update was at 11:03 PM ----------

Quote:
Originally Posted by Chubler_XL
Can you take note of how many lines are in the logfile when the process starts and then look for the "Application deployment complete" message on a line greater than this?
I theoretically could, but I think that's a less reliable way to work in as a condition -- it is possible for a deployment activity here to be triggered manually outside the context of the actions my script is taking, which wouldn't necessarily interfere with the script's actions, but would mess up the deployment 'instance' counting if I did it this way, particularly considering the variable length and history of the application's log. Good idea, just impractical in this particular scenario -- I appreciate it, though!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find time difference based on logfile

Hi All, Firstly thank you for the forum members I need to find time difference b'w two rows of timestamp using awk/shell. Here is the logfile: cat business_file start:skdjh:22:06:2010:10:30:22 sdfnskjoeirg wregn'wergnoeirnfqoeitgherg end:siifneworigo:22:06:2010:10:45:34... (3 Replies)
Discussion started by: Srinivas Gadi
3 Replies

2. Shell Programming and Scripting

Calculating the running time

Hi All, I want to run a utility for all the process id that are running for more than 15 mins. I have captured process id's and the time that they were run in a file like below 1st column represnts the process ids and the 2nd one is the Time < 21014 01:00 21099 01:00 24361 01:03 24406... (5 Replies)
Discussion started by: r_t_1601
5 Replies

3. Shell Programming and Scripting

Compare two timestamps and print elapsed time

Hi, I am unable to Difference between two time stamps in Linux and display the total elapsed time . Source date: Aug 15, 2012 02:00:03 Target date: Aug 14, 2012 18:00:03 # based on the forums I am using the below function. Converted dates into this format Src_dt=20120814180003... (7 Replies)
Discussion started by: onesuri
7 Replies

4. Shell Programming and Scripting

Calculating the epoch time from standard time using awk and calculating the duration

Hi All, I have the following time stamp data in 2 columns Date TimeStamp(also with milliseconds) 05/23/2012 08:30:11.250 05/23/2012 08:30:15.500 05/23/2012 08:31.15.500 . . etc From this data I need the following output. 0.00( row1-row1 in seconds) 04.25( row2-row1 in... (5 Replies)
Discussion started by: ks_reddy
5 Replies

5. UNIX for Dummies Questions & Answers

Calculating the Hours between two time values

Dear Folks, I want to calculate the elapsed hours between two time columns. I am using timestampdiff method for the same. I am able to get the value. But facing an issue of decimal values. For example the elapsed hours between 09:00:00 and 20:30:00 is coming as 11 instead of 11.5. I am using below... (1 Reply)
Discussion started by: dinesh1985
1 Replies

6. AIX

How to find time difference between 2 timestamps?

HI All, can some one please help me how to fine the difference between two time stamps say a= Nov 10, 2009 9:21:25 AM b= Nov 10, 2009 10:21:25 AM I want to find difference between the a & b I googled and tried with some options but no luck. My OS is AIX (1 Reply)
Discussion started by: bandlan9
1 Replies

7. Shell Programming and Scripting

Calculating completion time

The date construct in UNIX can be used to calculate when something is finished: date -v+1H displays the time 1 hour from now. I want to use the same construct in a script, but it is leading to error messages: echo "Finished at: " `date -v+$durationH` where $duration is calculated based on input... (3 Replies)
Discussion started by: figaro
3 Replies

8. Shell Programming and Scripting

Calculating delay time - bash

Hi, I am having the following problem. test > hourOfDay=06 ; delayTime=$(((9-$hourOfDay)*60)) ; echo $delayTime 180 test > hourOfDay=07 ; delayTime=$(((9-$hourOfDay)*60)) ; echo $delayTime 120 test > hourOfDay=08 ; delayTime=$(((9-$hourOfDay)*60)) ; echo $delayTime bash: (9-08: value... (5 Replies)
Discussion started by: jbsimon000
5 Replies

9. UNIX for Advanced & Expert Users

Caluculate time between 2 unix timestamps?

Does anyone know a perl script/utility/etc. to calculate the time in seconds between two unix time stamps? Any help will be appreciated. -bozzhawg (1 Reply)
Discussion started by: bozzhawg
1 Replies

10. UNIX for Dummies Questions & Answers

DiffElapsed time between unix timestamps help needed...

Does anyone know a perl script/utility/etc. to calculate the time in seconds between two unix time stamps? Any help will be appreciated. -bozzhawg (1 Reply)
Discussion started by: bozzhawg
1 Replies
Login or Register to Ask a Question