Sponsored Content
Top Forums Shell Programming and Scripting Extracting log entries from a date onwards Post 302511680 by rijeshpp on Thursday 7th of April 2011 10:17:35 AM
Old 04-07-2011
Extracting log entries from a date onwards

One of the log file looks like entries as below.

Code:
Wed Apr  6 14:51:18 2011 [pid 24366] [wasadmin] FAIL LOGIN: Client "9.191.21.54"
Wed Apr  6 14:52:53 2011 [pid 25554] CONNECT: Client "9.191.21.54"
Wed Apr  6 14:52:54 2011 [pid 25553] [wasadmin] OK LOGIN: Client "9.191.21.54"
Wed Apr  6 14:55:10 2011 [pid 27442] CONNECT: Client "9.191.21.54"
Wed Apr  6 14:55:12 2011 [pid 27441] [wasadmin] FAIL LOGIN: Client "9.191.21.54"
Wed Apr  6 14:56:12 2011 [pid 28291] CONNECT: Client "9.191.21.54"
Wed Apr  6 14:56:13 2011 [pid 28290] [wasadmin] OK LOGIN: Client "9.191.21.54"
Wed Apr  6 15:11:57 2011 [pid 8774] CONNECT: Client "9.191.21.54"
Wed Apr  6 15:11:57 2011 [pid 8773] [wasadmin] OK LOGIN: Client "9.191.21.54"
Thu Apr  7 19:02:54 2011 [pid 18139] CONNECT: Client "9.122.71.33"
Thu Apr  7 19:03:00 2011 [pid 18138] [wasadmin] FAIL LOGIN: Client "9.122.71.33"
Thu Apr  7 19:03:09 2011 [pid 18146] CONNECT: Client "9.122.71.33"
Thu Apr  7 19:03:15 2011 [pid 18145] [rijesh] OK LOGIN: Client "9.122.71.33"
Thu Apr  7 19:03:21 2011 [pid 18484] CONNECT: Client "9.122.71.33"
Thu Apr  7 19:03:24 2011 [pid 18483] [wasadmin] OK LOGIN: Client "9.122.71.33"
Thu Apr  7 19:03:29 2011 [pid 18492] CONNECT: Client "9.122.71.33"
Thu Apr  7 19:03:35 2011 [pid 18491] [wasadmin] FAIL LOGIN: Client "9.122.71.33"

From the above file, I want to develop a shell script that extract all lines lines from the above log file from the position onwards last time it executed(The last execution time will be recorded). For instance the log file may have thousands of line; the intention is to run the shell script as cronjob with a frequency of 15 minutes. So when the first time cronjob runs it will read all the files and the subsequent execution it needs to extract the lines of log entries from and after the time of script execution.

Having said the above requirement; I am looking for one piece of information here in this forum. I can get the execution time of the script in time format lets say using "date '+%T'". For example if I am executing script (cronjob) in the time of 19:00:00; then extract all the lines using grep that has entries after logged in 19:00:00 .I can have a logic of reading the log file line by line by incorporating a logic of incrementing execution time. However I suspect that may make the operation expensive if the log file has too many entries. So I am looking for a logic that is competitively less expensive operation. Any advice in this regards would be great help.

Thanks in advance,
Rijesh.

Last edited by vbe; 04-07-2011 at 11:20 AM.. Reason: typos: missng end to decode tag
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl: Extracting date from file name and comparing with current date

I need to extract the date part from the file name (20080221 in this ex) and compare it with the current date and delete it, if it is a past date. $file = exp_ABCD4_T-2584780_upto_20080221.dmp.Z really appreciate any help. thanks mkneni (4 Replies)
Discussion started by: MKNENI
4 Replies

2. UNIX for Dummies Questions & Answers

extracting date from a filename

Hi, I am a beginner in Unix so please bear with me... I have a directory which has files in format: RECF-YYYY-MM-DD-input. For example, RECF-2008-02-25-input. I need to extract the YYYYY-MM-DD substring from this filename and convert that into date and compare it with a date. How do I do that?... (7 Replies)
Discussion started by: laiko
7 Replies

3. Shell Programming and Scripting

Extracting Date from string

Hi Gurus I want to extract a date and version code which shall come in filename consisting of underscores. The filename can contain any / one underscores but the version number will come after date and will be separted by underscore String formats ============= ABC_20090815_2.csv... (13 Replies)
Discussion started by: r_t_1601
13 Replies

4. Shell Programming and Scripting

Extracting data from two date entries

Hi again: I have this file: "2010-11-1 11:50:00",40894,13.38,17.24,12.92,13.23,"2010-11-14 11:43:02",12.56,"2010-11-14 11:46:02",22.68,20.95,"2010-11-14 11:44:03",2.144,2.078,190.4,14.27,6.293,"2010-11-14 ... (2 Replies)
Discussion started by: iga3725
2 Replies

5. Homework & Coursework Questions

extracting date from log file

You are given a 1 year logfile with each line starting with a date in the form “YYYY-MM-DD”. How would you extract logs from the 4th day of each month and put them into a new file (1 Reply)
Discussion started by: DOkuwa
1 Replies

6. Shell Programming and Scripting

Extracting data from a log file with date formats

Hello, I have a log file for the year, which contains lines starting with the data in the format of YYYY-MM-DD. I need to get all the lines that contain the DD being 04, how would I do this? I tried using grep "*-*04" but it didn't work. Any quick one liners I should know about? Thank you. (2 Replies)
Discussion started by: cpickering
2 Replies

7. Shell Programming and Scripting

Need to check a file from a certain position and date onwards

Hi Guys, I need some advice please. My script is not grabbing information from a text file from a certain date correctly. It seems to be grabbing everying in the file, i know it is something simple but i have looked to hard and to long, to know what the issue is. Script awk '... (9 Replies)
Discussion started by: Junes
9 Replies

8. Shell Programming and Scripting

Delete log files content older than 30 days and append the lastest date log file date

To delete log files content older than 30 days and append the lastest date log file date in the respective logs I want to write a shell script that deletes all log files content older than 30 days and append the lastest log file date in the respective logs This is my script cd... (2 Replies)
Discussion started by: sreekumarhari
2 Replies

9. Shell Programming and Scripting

Monitor log entries in log files with no Date format? - Efficient logcheck?

is there a way to efficiently monitor logfiles that do not have a date or time format? i have several logs on several different servers that need to be monitored. but i realized writing a script for this would be very complex and time consuming giving the variety of things i need to check for i.e.... (2 Replies)
Discussion started by: SkySmart
2 Replies

10. Shell Programming and Scripting

Extracting log files based on date and time.

Hi All, i have some log files generated in a folder daily with the format abc.def.20130306.100001 ghi.jkl.20130306.100203 abc.def.20130305.100001 ghi.jkl.20130305.100203 the format is the date followed by time . all i want is to get the files that are generated for todays... (3 Replies)
Discussion started by: mahesh300182
3 Replies
XINETD.LOG(5)							File Formats Manual						     XINETD.LOG(5)

NAME
xinetd.log - xinetd service log format DESCRIPTION
A service configuration may specify various degrees of logging when attempts are made to access the service. When logging for a service is enabled, xinetd will generate one-line log entries which have the following format (all entries have a timestamp as a prefix): entry: service-id data The data depends on the entry. Possible entry types include: START generated when a server is started EXIT generated when a server exits FAIL generated when it is not possible to start a server USERID generated if the USERID log option is used. NOID generated if the USERID log option is used, and the IDONLY service flag is used, and the remote end does not identify who is trying to access the service. In the following, the information enclosed in brackets appears if the appropriate log option is used. A START entry has the format: START: service-id [pid=%d] [from=%d.%d.%d.%d] An EXIT entry has the format: EXIT: service-id [type=%d] [pid=%d] [duration=%d(sec)] type can be either status or signal. The number is either the exit status or the signal that caused process termination. A FAIL entry has the format: FAIL: service-id reason [from=%d.%d.%d.%d] Possible reasons are: fork a certain number of consecutive fork attempts failed (this number is a configurable parameter) time the time check failed address the address check failed service_limit the allowed number of server instances for this service would be exceeded process_limit a limit on the number of forked processes was specified and it would be exceeded A DATA entry has the format: DATA: service-id data The data logged depends on the service. login remote_user=%s local_user=%s tty=%s exec remote_user=%s verify=status command=%s Possible status values: ok the password was correct failed the password was incorrect baduser no such user shell remote_user=%s local_user=%s command=%s finger received string or EMPTY-LINE A USERID entry has the format: USERID: service-id text The text is the response of the identification daemon at the remote end excluding the port numbers (which are included in the response). A NOID entry has the format: NOID: service-id IP-address reason SEE ALSO
xinetd(1L), xinetd.conf(5) 28 April 1993 XINETD.LOG(5)
All times are GMT -4. The time now is 08:30 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy