Fetching timestamps from the logs.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Fetching timestamps from the logs.
# 1  
Old 01-18-2018
Fetching timestamps from the logs.

Dear Experts,

I need some help to get the time stamps from the Unix logs. Basically I am want to analyze which step is taking more time. Please give me some inputs. Thanks a lot for your help.

I have got different codes for each for phase as we can see in the logs
Code:
00002 – UPDT Starting
00008 – UPDT ending
00199 – TPLD Starting
00907 – TPLD Ending


Input:

Code:
01182018 095440 Prod TPLD test_system-00199-Information (Z0000001) Begin FastLoad for table Z0000001 on Prod XXXXX5.
01182018 100009 Prod TPLD test_system-00199-Information (Z0000001) FastLoad for table Z0000001 on Prod XXXXX5 completed.
01182018 100010 Prod TPLD test_system-00907-Information (Z0000001) 17403078 records loaded to Z0000001 on Prod XXXXX5.
01182018 101127 Prod TPLD test_system-00902-Information (Z0000001) Finished Temporary Load Phase (Z0000001)
01182018 101324 Prod UPDT test_system-00901-Information (Z0000001) Started Update Phase (Z0000001)
01182018 101332 Prod UPDT test_system-00002-Information (Z0000001) Started OTHER (01) on table sample_table (Prod XXXXX5)
01182018 101336 Prod UPDT test_system-00008-Information (Z0000001) CIH/other_IH92_1.sql script successfull on sample_table table on
Prod XXXXX5
01182018 101340 Prod UPDT test_system-00002-Information (Z0000001) Started OTHER (02) on table sample_table (Prod XXXXX5)
01182018 102530 Prod UPDT test_system-00008-Information (Z0000001) CIH/other_IH92_2.sql script successfull on sample_table table on
Prod XXXXX5
01182018 102626 Prod UPDT test_system-00002-Information (Z0000001) Started INSERT (03) on table sample_table (Prod XXXXX5)
01182018 102959 Prod UPDT test_system-00008-Information (Z0000001) 17403694 record(s) INSERTed (03) into sample_table table on Prod
XXXXX5
01182018 112646 Prod UPDT test_system-00902-Information (Z0000001) Finished Update Phase (Z0000001)


Desired output:

Code:
FILE_NBR|PHASE|START_TIME|END_TIME|STEP
Z0000001|TPLD|01182018 095440|01182018 100010|NA
Z0000001|UPDT|01182018 101332|01182018 101336|01
Z0000001|UPDT|01182018 101340|01182018 102530|02
Z0000001|UPDT|01182018 102626|01182018 102959|03


Last edited by srikanth38; 01-19-2018 at 11:04 AM.. Reason: Add missing CODE tags.
# 2  
Old 01-19-2018
Quote:
Originally Posted by srikanth38
Dear Experts,

I need some help to get the time stamps from the Unix logs. Basically I am want to analyze which step is taking more time. Please give me some inputs. Thanks a lot for your help.

I have got different codes for each for phase as we can see in the logs
Code:
00002 - UPDT Starting
00008 - UPDT ending
00199 - TPLD Starting
00907 - TPLD Ending


Input:

Code:
01182018 095440 Prod TPLD test_system-00199-Information (Z0000001) Begin FastLoad for table Z0000001 on Prod XXXXX5.
01182018 100009 Prod TPLD test_system-00199-Information (Z0000001) FastLoad for table Z0000001 on Prod XXXXX5 completed.
01182018 100010 Prod TPLD test_system-00907-Information (Z0000001) 17403078 records loaded to Z0000001 on Prod XXXXX5.
01182018 101127 Prod TPLD test_system-00902-Information (Z0000001) Finished Temporary Load Phase (Z0000001)
01182018 101324 Prod UPDT test_system-00901-Information (Z0000001) Started Update Phase (Z0000001)
01182018 101332 Prod UPDT test_system-00002-Information (Z0000001) Started OTHER (01) on table sample_table (Prod XXXXX5)
01182018 101336 Prod UPDT test_system-00008-Information (Z0000001) CIH/other_IH92_1.sql script successfull on sample_table table on
Prod XXXXX5
01182018 101340 Prod UPDT test_system-00002-Information (Z0000001) Started OTHER (02) on table sample_table (Prod XXXXX5)
01182018 102530 Prod UPDT test_system-00008-Information (Z0000001) CIH/other_IH92_2.sql script successfull on sample_table table on
Prod XXXXX5
01182018 102626 Prod UPDT test_system-00002-Information (Z0000001) Started INSERT (03) on table sample_table (Prod XXXXX5)
01182018 102959 Prod UPDT test_system-00008-Information (Z0000001) 17403694 record(s) INSERTed (03) into sample_table table on Prod
XXXXX5
01182018 112646 Prod UPDT test_system-00902-Information (Z0000001) Finished Update Phase (Z0000001)


Desired output:

Code:
FILE_NBR|PHASE|START_TIME|END_TIME|STEP
Z0000001|TPLD|01182018 095440|01182018 100009|NA
Z0000001|UPDT|01182018 101332|01182018 101336|01
Z0000001|UPDT|01182018 101340|01182018 102530|02
Z0000001|UPDT|01182018 102626|01182018 102959|03

What operating system are you using?

What shell are you using?

What have you tried to solve this problem on your own?

Given that you have two "TPLD Starting" events at the start of your log file (marked above in red) and one "TPLD Ending" event (marked in blue), why are the timestamps on the TPLD activity in your output from the two Starting events instead of the "START_TIME" coming from one of the Starting events and the "END_TIME" coming from the Ending event? If you have two starting events with only one ending event, why did you choose to use the first starting event timestamp instead of the second starting event timestamp? (I.e., if the activity was restarted, why not choose the restart time instead of the time that never completed?)
This User Gave Thanks to Don Cragun For This Post:
# 3  
Old 01-19-2018
Quote:
Originally Posted by Don Cragun
What operating system are you using?

What shell are you using?

What have you tried to solve this problem on your own?

Given that you have two "TPLD Starting" events at the start of your log file (marked above in red) and one "TPLD Ending" event (marked in blue), why are the timestamps on the TPLD activity in your output from the two Starting events instead of the "START_TIME" coming from one of the Starting events and the "END_TIME" coming from the Ending event? If you have two starting events with only one ending event, why did you choose to use the first starting event timestamp instead of the second starting event timestamp? (I.e., if the activity was restarted, why not choose the restart time instead of the time that never completed?)
I tried grep the lines for every phase but its becoming a complex program as I had different other phases as well.
I am using Linux and korn shell. Anytime 01182018 100009, 01182018 100010 works for me to know TPLD completion. 01182018 100009 more accurate, since 01182018 100010 has a different code I have choosed.
# 4  
Old 01-19-2018
Quote:
I tried grep the lines ......
This doesn't actually tell us what you have tried. Can you post your attempts and output/errors into the thread wrapped in CODE tags please. Then we might be able to make suggestions.




Kind regards,
Robin

Last edited by rbatte1; 01-19-2018 at 12:26 PM.. Reason: Typo correction
# 5  
Old 01-19-2018
In addition to what Robin has already said, you need to explain MUCH more clearly how you want to specify rules that govern the extraction of the start and end times for TPLD events. You gave ambiguous requirements in your first post, but your sample output doesn't even meet those requirements. You say that are are using "a different code that I have choosed (sic)."

I have an awk script that seems to meet your written requirements, but it doesn't work at all in producing your desired output for TPLD transactions; it only produces your desired output for UPDT transactions.
# 6  
Old 01-20-2018
Quote:
Originally Posted by Don Cragun
In addition to what Robin has already said, you need to explain MUCH more clearly how you want to specify rules that govern the extraction of the start and end times for TPLD events. You gave ambiguous requirements in your first post, but your sample output doesn't even meet those requirements. You say that are are using "a different code that I have choosed (sic)."

I have an awk script that seems to meet your written requirements, but it doesn't work at all in producing your desired output for TPLD transactions; it only produces your desired output for UPDT transactions.
Thanks for looking into it. I have corrected it later. Its my bad. I will try myself and let you know.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

If I ran perl script again,old logs should move with today date and new logs should generate.

Appreciate help for the below issue. Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt.............. 1)if logs exist it should move the logs with extention... (1 Reply)
Discussion started by: Sanjeev G
1 Replies

2. Shell Programming and Scripting

[Solved] Fetching logs of last few days from logfile

Hi All, I have a requirement to fetch logs of last 'N' days. I am trying the following command which is working fine if it finds the date of that day in logfile. START=`TZ="GMT+$((24*N))" date +"%Y %b %d"` this is being used to fetch 'N'th day's date and awk '/'"$START"'/{p=1}... (24 Replies)
Discussion started by: KDMishra
24 Replies

3. Shell Programming and Scripting

Compare Timestamps

Hi! Long time reader first time registered user and poster. I've picked up some times and tricks and I'm at a dead end... I've parsed a log file for duplicates and printed only the two fields I need (duplicate entry and time stamp). My question is, with this output, how would I script... (2 Replies)
Discussion started by: rexpokinghorn
2 Replies

4. Shell Programming and Scripting

Logs access in windows fetching the data from a unix server

How I can get the logs that are getting stored in specific location in unix server through an Apache web server installed in unix server? Requirement is to access the logs through the URL in windows browser without any access. (1 Reply)
Discussion started by: alvida
1 Replies

5. Shell Programming and Scripting

Arithmetic on timestamps

Hi Friends, please advise on shell script to add two time stamps for example : a=12:32 b=12:00 c=a+b=00:32 please help me to find shell script to add to two time stamps, as i need to convert time from EST to GMT or SST to prepare status of jobs in unix and to specify estimated time to... (3 Replies)
Discussion started by: balireddy_77
3 Replies

6. Shell Programming and Scripting

Strip timestamps from logs using sed?

Hullo. I have a log file: (I) 02/03/2009 12:31:01 : Service Started (I) 02/03/2009 12:31:02 : XML: <xml> <stuff> ... </xml> (I) 02/03/2009 12:31:02 : Service Stoped ... ...and I would like to remove date/time/PID-stamps so I can create a baseline for future comparisson. Ideal... (5 Replies)
Discussion started by: cs03dmj
5 Replies

7. Shell Programming and Scripting

Grep yesterday logs from weblogic logs

Hi, I am trying to write a script which would go search and get the info from the logs based on yesterday timestamp and write yesterday logs in new file. The log file format is as follows: """"""""""""""""""""""""""... (3 Replies)
Discussion started by: harish.parker
3 Replies

8. Shell Programming and Scripting

timestamps

Hello! I have the following problem. I read a file using perl, each line of this file has the fllowing format. 14/4/2008 8:42:03 πμ|10800|306973223399|4917622951117|1||1259|1|126|492|433||19774859454$ Th first field is the timestamp and the second field is the offset in seconds. How can... (1 Reply)
Discussion started by: chriss_58
1 Replies

9. Solaris

Difference between two timestamps

I'm writting a script to find the difference between two timestamp. One field i get on delivery time of the file like 07:17 AM and other is my SLA time 06:30 AM I need to find the difference between these two time (time exceeded to meet SLA). Need some suggestions. (8 Replies)
Discussion started by: raman1605
8 Replies

10. UNIX for Dummies Questions & Answers

Unix timestamps

Can someone help me with a Unix or perl script to convert the unix timestamps to human readable format? Any help will be highly appreciated... (3 Replies)
Discussion started by: hamsasal
3 Replies
Login or Register to Ask a Question