Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extracting the last 10mins worth of data in a log file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extracting the last 10mins worth of data in a log file
# 1  
Old 05-16-2012
Extracting the last 10mins worth of data in a log file
Hi all,
Hope someone here will be able to help me. Our system has some scripts that are run from a cron job every ten mins and is used to see how many error there are in that time frame. Problem is that in the scripts grep is used to find the data, but as the day goes on these log file grow to a huge size and the whole process is getting slower and slower.

What I need to do if figure out a smarter way of doing it. So what I was looking for is there some way I can extract the last 10 mins (even have this variable) worth of data from the logs, with out having to scan through the whole file.

I was think maybe of reading the file from the bottom up to 10 mins ago, some thing like the tac command, but the problem with the tac command is it will read the whole file which I want to avoid.

All the entries in the log file are timestamped in the following format:
2012-05-16 17:00:46

I am on a linuix system and using ksh.
Scripting is not a strong point of mine
Thanks all in advance.
# 2  
Old 05-16-2012
I'm presuming you have GNU date since you have Linux so you can do
Code:
date -d " - 10 minutes" +"%Y-%m-%d %H:%M:%S"

An interesting property of YYYY-MM-DD HH:MM:SS timestamps is they sort and compare in proper order. Later dates will be greater than earlier dates in simple ASCII < > comparison.

Code:
awk -v LAST=$(date -d " - 10 minutes" +"%Y-%m-%d %H:%M:%S") '($1 " " $2) >= LAST' logfile

If that doesn't work, please post a few lines of your logfile so I can test my solutions before posting them.
# 3  
Old 05-16-2012
Here is an extract from the log file:
Code:
2012-05-16 17:00:38 AA AAFS;
2012-05-16 17:00:38 BB AAFS;
2012-05-16 17:00:38 CC AAFS;
2012-05-16 17:00:39 DD AAFS;
2012-05-16 17:00:39 EE AACS;
2012-05-16 17:00:39 FF AAFS;
2012-05-16 17:00:39 GG AAFS;
2012-05-16 17:00:40 HH AAFS;
2012-05-16 17:00:40 II AAFS;
2012-05-16 17:00:41 JJ AAFS;
2012-05-16 17:00:42 KK AAFS;
2012-05-16 17:00:42 LL AAFS;
2012-05-16 17:00:42 MM AAFS;
2012-05-16 17:00:43 NN AAFS;
2012-05-16 17:00:45 OO AACS;
2012-05-16 17:00:46 PP AAFS;
2012-05-16 17:00:46 QQ AAFS;

when there is an error it will be printer out like this :
Code:
2012-05-16 17:00:46 QQ AAFS [ERROR] Some error;

When i run the command

Code:
awk -v LAST=$(date -d "-10 minutes" +"%Y-%m-%d %H:%M:%S") '($1 " " $2) >= LAST' logfile

I get a parse error:
Code:
awk: cmd. line:1: 17:38:58
awk: cmd. line:1:   ^ parse error

17:38:58 is the current time - 10 mins

(ps had to delete some spaces in your command as it was adding ten mins to the current time rather that taking away tens mins)
# 4  
Old 05-16-2012
There was a minor mistake in my script sorry, I forgot to quote the variable there and it was splitting. I didn't think $( ) did split, I find myself corrected...

Code:
awk -v LAST="$(date -d "-10 minutes" +"%Y-%m-%d %H:%M:%S")" '($1 " " $2) >= LAST' logfile

# 5  
Old 05-17-2012
Corona688 thanks for the help so far.

I made a mistake here in the format of the file I am looking at, the format is like this and not what I posted before:


Code:
[2012-05-17 00:01:27,937] [ INFO] - - 
[2012-05-17 00:01:27,952] [ INFO] - - 
[2012-05-17 00:01:28,019] [ INFO] - - 
[2012-05-17 00:01:28,150] [ INFO] - - 
[2012-05-17 00:01:28,151] [ INFO] - - 
[2012-05-17 00:01:28,183] [ INFO] - - 
[2012-05-17 00:01:28,281] [ INFO] - - 
[2012-05-17 00:01:28,519] [ INFO] - - 
[2012-05-17 00:01:28,519] [ INFO] - - 
[2012-05-17 00:01:28,556] [ INFO] - - 
[2012-05-17 00:01:28,615] [ INFO] - - 
[2012-05-17 00:01:28,934] [ INFO] - - 
[2012-05-17 00:01:28,934] [ INFO] - - 
[2012-05-17 00:01:28,969] [ INFO] - - 
[2012-05-17 00:01:29,053] [ INFO] - - 
[2012-05-17 00:01:29,601] [ ERROR] - -

Moderator's Comments:
Mod Comment Video tutorial on how to use code tags in The UNIX and Linux Forums.
# 6  
Old 05-17-2012
Okay. That's still ASCIIbetical YYYY MM DD HH MM SS so just a little more wrangling to get the exact text to be compared.

Code:
awk -F"," -v LAST="$(date -d "-10 minutes" +"%Y-%m-%d %H:%M:%S")" 'substr($1, 2) >= LAST' logfile

This User Gave Thanks to Corona688 For This Post:
Goldengreen
# 7  
Old 05-17-2012
Ah yes getting places now :-)

Looks to be extracting correctly now.

One question though will awk scan through the file from top to bottom, is it possible to get it to scan from the bottom up and once it has all ten mins worth of data stop at the point.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extracting data from rsync log

Hi I have a daily rsync that runs and i am trying to find a easy way of extracting the start time and end time of the sync and extract the details of how much data was copied. I would like to use this information to see if i can increase the amount of rsyncs that run in a day. so the log... (3 Replies)
Discussion started by: treds
3 Replies

2. Shell Programming and Scripting

Error while extracting data from log file

I am running awk command to extract data from log file to calculate last 15 minutes log using below command and now i am getting bellow error: awk '$0>=$from' from=$(`date -u +"####<%d-%b-%Y %H:%M:%S o'clock GMT>"-15min`) test.log Error: date: 0551-402 Invalid character in date/time... (8 Replies)
Discussion started by: oberoi1403
8 Replies

3. Shell Programming and Scripting

Need Help in extracting data from XML File

Hi All My input file is an XML and it has some tags and data rows at end. Starting of data rows is <rs:data> and ending of data rows is </rs:data>. Within sample data rows (2 rows) shown below, I want to extract data value after equal to sign (until space or "/" sign). So if XML data... (7 Replies)
Discussion started by: vx04
7 Replies

4. Shell Programming and Scripting

Extracting Delimiter 'TAG' Data From log files

Hi I am trying to extract data from within a log file and output format to a new file for further manipulation can someone provide script to do this? For example I have a file as below and just want to extract all delimited variances of tag 32=* up to the delimiter "|" and output to a new file... (2 Replies)
Discussion started by: Buddyluv
2 Replies

5. UNIX for Dummies Questions & Answers

Extracting data from file

I am trying to compare the data in lines 3 & 5 to see if they match up to the '-S570' (see first code set, all proprietary information has been removed from code set) spawn telnet Trying ... Connected to CA-LOS1234-ASE-S570.cl . Escape character is '^]'. CA-LOS1234-ASE-S570 Username: ... (1 Reply)
Discussion started by: slipshft
1 Replies

6. Shell Programming and Scripting

Extracting specific lines of data from a file and related lines of data based on a grep value range?

Hi, I have one file, say file 1, that has data like below where 19900107 is the date, 19900107 12 144 129 0.7380047 19900108 12 168 129 0.3149017 19900109 12 192 129 3.2766666E-02 ... (3 Replies)
Discussion started by: Wynner
3 Replies

7. Shell Programming and Scripting

Extracting data from a log file with date formats

Hello, I have a log file for the year, which contains lines starting with the data in the format of YYYY-MM-DD. I need to get all the lines that contain the DD being 04, how would I do this? I tried using grep "*-*04" but it didn't work. Any quick one liners I should know about? Thank you. (2 Replies)
Discussion started by: cpickering
2 Replies

8. Shell Programming and Scripting

extracting data from a .csv file

I have a .csv file equipment,bandtype abc,aws def,mmds ghi,umts jkl,mmds I can get the equipment from `hostname`. In my script i want to check what is the hostname. then see if it exists in the.csv file. if it does then i want to store the second parameter(bandtype) for the corresponding... (3 Replies)
Discussion started by: lassimanji
3 Replies

9. Shell Programming and Scripting

How do i get only last 5 minute worth of data

I have a text file called 'tomcat_temp_out'. I want to get only last 5 minute worth of data from this file and redirect those data into another fule. Could you please help to work on this? (2 Replies)
Discussion started by: shivanete
2 Replies

10. UNIX for Dummies Questions & Answers

Extracting Data from a File

Hi I need to calculate the number of occurrences of a item in a number of files using Perl. The item appears continually throughout the files but in each case I only want to calculate it in certain blocks of the file. Example - Calculalte the number of occurrences of a 'pass' in a block of... (0 Replies)
Discussion started by: oop
0 Replies
Login or Register to Ask a Question