Extracting log entries from a date onwards


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extracting log entries from a date onwards
# 1  
Old 04-07-2011
Extracting log entries from a date onwards

One of the log file looks like entries as below.

Code:
Wed Apr  6 14:51:18 2011 [pid 24366] [wasadmin] FAIL LOGIN: Client "9.191.21.54"
Wed Apr  6 14:52:53 2011 [pid 25554] CONNECT: Client "9.191.21.54"
Wed Apr  6 14:52:54 2011 [pid 25553] [wasadmin] OK LOGIN: Client "9.191.21.54"
Wed Apr  6 14:55:10 2011 [pid 27442] CONNECT: Client "9.191.21.54"
Wed Apr  6 14:55:12 2011 [pid 27441] [wasadmin] FAIL LOGIN: Client "9.191.21.54"
Wed Apr  6 14:56:12 2011 [pid 28291] CONNECT: Client "9.191.21.54"
Wed Apr  6 14:56:13 2011 [pid 28290] [wasadmin] OK LOGIN: Client "9.191.21.54"
Wed Apr  6 15:11:57 2011 [pid 8774] CONNECT: Client "9.191.21.54"
Wed Apr  6 15:11:57 2011 [pid 8773] [wasadmin] OK LOGIN: Client "9.191.21.54"
Thu Apr  7 19:02:54 2011 [pid 18139] CONNECT: Client "9.122.71.33"
Thu Apr  7 19:03:00 2011 [pid 18138] [wasadmin] FAIL LOGIN: Client "9.122.71.33"
Thu Apr  7 19:03:09 2011 [pid 18146] CONNECT: Client "9.122.71.33"
Thu Apr  7 19:03:15 2011 [pid 18145] [rijesh] OK LOGIN: Client "9.122.71.33"
Thu Apr  7 19:03:21 2011 [pid 18484] CONNECT: Client "9.122.71.33"
Thu Apr  7 19:03:24 2011 [pid 18483] [wasadmin] OK LOGIN: Client "9.122.71.33"
Thu Apr  7 19:03:29 2011 [pid 18492] CONNECT: Client "9.122.71.33"
Thu Apr  7 19:03:35 2011 [pid 18491] [wasadmin] FAIL LOGIN: Client "9.122.71.33"

From the above file, I want to develop a shell script that extract all lines lines from the above log file from the position onwards last time it executed(The last execution time will be recorded). For instance the log file may have thousands of line; the intention is to run the shell script as cronjob with a frequency of 15 minutes. So when the first time cronjob runs it will read all the files and the subsequent execution it needs to extract the lines of log entries from and after the time of script execution.

Having said the above requirement; I am looking for one piece of information here in this forum. I can get the execution time of the script in time format lets say using "date '+%T'". For example if I am executing script (cronjob) in the time of 19:00:00; then extract all the lines using grep that has entries after logged in 19:00:00 .I can have a logic of reading the log file line by line by incorporating a logic of incrementing execution time. However I suspect that may make the operation expensive if the log file has too many entries. So I am looking for a logic that is competitively less expensive operation. Any advice in this regards would be great help.

Thanks in advance,
Rijesh.

Last edited by vbe; 04-07-2011 at 11:20 AM.. Reason: typos: missng end to decode tag
# 2  
Old 04-07-2011
Could you post what you have done so far?
# 3  
Old 04-07-2011
You could try something like this to get the time stamps into a Unix format. Then compare that to the stored value.
Code:
while read LINE; do
  DATESTR=$(echo $LINE | cut -b -23)
  TIME=$(date -d "$DATESTR" +%s)
  echo $TIME
done

It could get time consuming on a large file though...

Last edited by linuxhitman; 04-07-2011 at 06:34 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extracting log files based on date and time.

Hi All, i have some log files generated in a folder daily with the format abc.def.20130306.100001 ghi.jkl.20130306.100203 abc.def.20130305.100001 ghi.jkl.20130305.100203 the format is the date followed by time . all i want is to get the files that are generated for todays... (3 Replies)
Discussion started by: mahesh300182
3 Replies

2. Shell Programming and Scripting

Monitor log entries in log files with no Date format? - Efficient logcheck?

is there a way to efficiently monitor logfiles that do not have a date or time format? i have several logs on several different servers that need to be monitored. but i realized writing a script for this would be very complex and time consuming giving the variety of things i need to check for i.e.... (2 Replies)
Discussion started by: SkySmart
2 Replies

3. Shell Programming and Scripting

Delete log files content older than 30 days and append the lastest date log file date

To delete log files content older than 30 days and append the lastest date log file date in the respective logs I want to write a shell script that deletes all log files content older than 30 days and append the lastest log file date in the respective logs This is my script cd... (2 Replies)
Discussion started by: sreekumarhari
2 Replies

4. Shell Programming and Scripting

Need to check a file from a certain position and date onwards

Hi Guys, I need some advice please. My script is not grabbing information from a text file from a certain date correctly. It seems to be grabbing everying in the file, i know it is something simple but i have looked to hard and to long, to know what the issue is. Script awk '... (9 Replies)
Discussion started by: Junes
9 Replies

5. Shell Programming and Scripting

Extracting data from a log file with date formats

Hello, I have a log file for the year, which contains lines starting with the data in the format of YYYY-MM-DD. I need to get all the lines that contain the DD being 04, how would I do this? I tried using grep "*-*04" but it didn't work. Any quick one liners I should know about? Thank you. (2 Replies)
Discussion started by: cpickering
2 Replies

6. Homework & Coursework Questions

extracting date from log file

You are given a 1 year logfile with each line starting with a date in the form “YYYY-MM-DD”. How would you extract logs from the 4th day of each month and put them into a new file (1 Reply)
Discussion started by: DOkuwa
1 Replies

7. Shell Programming and Scripting

Extracting data from two date entries

Hi again: I have this file: "2010-11-1 11:50:00",40894,13.38,17.24,12.92,13.23,"2010-11-14 11:43:02",12.56,"2010-11-14 11:46:02",22.68,20.95,"2010-11-14 11:44:03",2.144,2.078,190.4,14.27,6.293,"2010-11-14 ... (2 Replies)
Discussion started by: iga3725
2 Replies

8. Shell Programming and Scripting

Extracting Date from string

Hi Gurus I want to extract a date and version code which shall come in filename consisting of underscores. The filename can contain any / one underscores but the version number will come after date and will be separted by underscore String formats ============= ABC_20090815_2.csv... (13 Replies)
Discussion started by: r_t_1601
13 Replies

9. UNIX for Dummies Questions & Answers

extracting date from a filename

Hi, I am a beginner in Unix so please bear with me... I have a directory which has files in format: RECF-YYYY-MM-DD-input. For example, RECF-2008-02-25-input. I need to extract the YYYYY-MM-DD substring from this filename and convert that into date and compare it with a date. How do I do that?... (7 Replies)
Discussion started by: laiko
7 Replies

10. Shell Programming and Scripting

Perl: Extracting date from file name and comparing with current date

I need to extract the date part from the file name (20080221 in this ex) and compare it with the current date and delete it, if it is a past date. $file = exp_ABCD4_T-2584780_upto_20080221.dmp.Z really appreciate any help. thanks mkneni (4 Replies)
Discussion started by: MKNENI
4 Replies
Login or Register to Ask a Question