Best approach for a 10 min extract out of several log files with timestamped records


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Best approach for a 10 min extract out of several log files with timestamped records
# 1  
Old 11-14-2005
Error Best approach for a 10 min extract out of several log files with timestamped records

I have a task where I need to code a shell script to extract a 10 min range (10 min from now until now) extract of a log file.

I taught I could simply use a command that would say something like Start=date - 10 min but I didn't find anything. Looks like the only solution would have to code a small script that would extract the time, strip the values, sub 10 min, check for month and year crossover and convert back into a date string.

Idealy, if there was a command to extract all records from start (chronologicaly reveresed) up until matching record= 10 min back, that would be great. I seriously dought there would be such a thing.

Any suggestions ? I'm a Unix script beginner.
# 2  
Old 11-14-2005
Post some sample data.
# 3  
Old 11-14-2005
Sample data

sample file to have 10 min extracted. It is the result of an fgrep done on several log files on specific keywords :

/export/home/..../filea.log:2005-11-14 12:01:03,999 xxxxxxxxxxxxxxx
/export/home/..../filea.log:2005-11-14 12:01:02,123 xxxxxxxxxxxxxxx
/export/home/..../filea.log:2005-11-14 12:01:01,345 xxxxxxxxxxxxxxx
...

The logs are live and therefore always updated.

Once I have the date-10min value, wouldn't it be possible to somehow get the line# of where the 10 min matches and have a command that only gets from the start up to that line# ?
# 4  
Old 11-15-2005
With only 10 minutes to look for, you could put all 10 timestamps in one grep statement. To reverse the lines, prepend line numbers, sort in reverse order on the line numbers, and remove the line numbers. This script uses datacalc which you can find on this site.

Code:
#! /usr/bin/ksh

date "+%Y %m %d %H %M" | read year month day hour minute
pattern=""
count=10
while((count)) ; do
        now="${year}-${month}-${day} ${hour}:${minute}"
        ((count=count-1))
        if ((minute)) ; then
                ((minute=minute-1))
        else
                minute=59
                if ((hour)) ; then
                        ((hour=hour-1))
                else
                        hour=23
                        datecalc -a $year $month $day - 1 | read year month day
                fi
        fi
        [[ -n $pattern ]] && pattern="${pattern}|"
        pattern="${pattern}${now}"
done
pattern=$(echo $pattern | tr "|" "\n")

grep -F "$pattern" | nl -ba | sort -nr | cut -f2-
exit 0

 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Extract lines if string found from last 30 min only

Hi guys, Appreciate your help as I am stuck with searching the logs for last 30 minutes from the current time. Current time is time when you execute the script and it will search for <string> through the logs for last 30 minutes only and if <string> found then print those lines only. The... (18 Replies)
Discussion started by: rockstar
18 Replies

2. Shell Programming and Scripting

How to extract part of string from all log files.?

Hi All, Let us say we have 5 log files, extract the data from all log files and save the output in a file. home/log/first.log home/log/second.log home/log/third.log home/log/four.log home/log/five.log I want to extract the following text from the log files and save the output in a file.... (7 Replies)
Discussion started by: ROCK_PLSQL
7 Replies

3. Shell Programming and Scripting

Extract lines with min value, using two field separators.

I have a file with two ID columns followed by five columns of counts in fraction form. I'd like to print lines that have a count of at least 4 (so at least 4 in the numerator, e.g. 4/17) in at least one of the five columns. Input file: comp51820_c1_seq1 693 0/29 0/50 0/69 0/36 0/31... (6 Replies)
Discussion started by: pathunkathunk
6 Replies

4. Shell Programming and Scripting

Can you extract (remove) lines from log files?

I use "MineOS" (a linux distro with python scripts and web ui included for managing a Minecraft Server). The author of the scripts is currently having a problem with the Minecraft server log file being spammed with certain entries. He's working on clearing up the spam. But in the meantime, I'm... (8 Replies)
Discussion started by: nbsparks
8 Replies

5. Shell Programming and Scripting

Extract records from list

Hi Gents, I have a file 1 like this 1 1000 20 2 2000 30 3 1000 40 5 1000 50 And I have other file 1 like 2 1 I would like to get from the file 1 the complete line which are in file 2, the key to compare is the column 2 then output should be. 2 2000 30. I was trying to get it... (5 Replies)
Discussion started by: jiam912
5 Replies

6. Shell Programming and Scripting

Compare two files with different number of records and output only the Extra records from file1

Hi Freinds , I have 2 files . File 1 |nag|HYd|1|Che |esw|Gun|2|hyd |pra|bhe|3|hyd |omu|hei|4|bnsj |uer|oeri|5|uery File 2 |nag|HYd|1|Che |esw|Gun|2|hyd |uer|oi|3|uery output : (9 Replies)
Discussion started by: i150371485
9 Replies

7. Shell Programming and Scripting

Creating timestamped directories/files in expect

Hi, I am looking for the syntax to provide a timestamped log_file in expect. Basically I want the equivalent of: /outputs/`date +%d%m%y`/session`date +%H%M` But scripted in expect so it can be handed over to the logfile function. I have tried playing around with the timestamp... (1 Reply)
Discussion started by: krypton
1 Replies

8. Shell Programming and Scripting

Compare Records between to files and extract it

I am not an expert in awk, SED, etc... but I really hope there is a way to do this, because I don't want to have to right a program. I am using C shell. FILE 1 FILE 2 H0000000 H0000000 MA1 MA1 CA1DDDDDD CA1AAAAAA MA2 ... (2 Replies)
Discussion started by: jclanc8
2 Replies

9. Shell Programming and Scripting

Need to Extract Data From 94000 records

i have a input file which does not have a delimiter All i Need to do is to identify a line and extract the data from it and run the loop again and need to ensure that it was not extracted earlier Input file ------------ abcd 12345 egfhijk ip 192.168.0.1 CNN.com abcd 12345 egfhijk ip... (12 Replies)
Discussion started by: vasimm
12 Replies

10. AIX

need to extract info from log files

hi guys i need to extract information from log files generated by an application. log file has the following lines for each process.. ---------------------------------------------- Fri Aug 03 12:06:43 WST 2007 INFO: Running project PROJECT1 Fri Aug 03 12:06:43 WST 2007 INFO: Source Files... (7 Replies)
Discussion started by: kirantalla
7 Replies
Login or Register to Ask a Question