Visit Our UNIX and Linux User Community


Filter records from a log file based on timestamp


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Filter records from a log file based on timestamp
# 1  
Old 01-20-2020
Filter records from a log file based on timestamp

Dear Experts,

I have a log file that contains a timestamp, I would like to filter record from that file based on timestamp. For example refer below file -
cat sample.txt
Code:
Jan 19 20:51:48 mukul-Vostro-14-3468 systemd: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Jan 19 20:54:48 mukul-Vostro-14-3468 su[806]: pam_unix(su:session): session closed for user root
Jan 19 20:57:48 mukul-Vostro-14-3468 systemd-logind[1048]: Removed session c49.
Jan 19 20:59:48 mukul-Vostro-14-3468 su[854]: Successful su for root by root
Jan 19 20:57:48 mukul-Vostro-14-3468 su[854]: + ??? root:root
Jan 19 21:05:48 mukul-Vostro-14-3468 su[854]: pam_unix(su:session): session opened for user root by (uid=0)
Jan 19 21:10:48 mukul-Vostro-14-3468 systemd-logind[1048]: New session c50 of user root.
Jan 19 21:12:48 mukul-Vostro-14-3468 systemd: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Jan 19 21:15:49 mukul-Vostro-14-3468 su[854]: pam_unix(su:session): session closed for user root

Out of this file i would like to filter record before 21:10 hrs.

Any help is appreciated.
# 2  
Old 01-20-2020
Any attempts / ideas / thoughts from your side? Did you consider searching tis site, or looking into the links at the bottom of this thread?


Are the entries sorted by timestamp?
# 3  
Old 01-20-2020
Hello Rudic,

Thanks for your quick revert. Yes the entries are sorted. Haven't gave it a shot since i wasn't sure where to start with. One thing that i got to know is that the timestamp need to be converted into string first before i apply any sort of comparison.
# 4  
Old 01-20-2020
OK; give it a shot and report back the errors and/or shortfalls you encounter.
This User Gave Thanks to RudiC For This Post:
# 5  
Old 01-29-2020
Dear Experts,
This is what I've tried by converting into epoch time and it's giving me the desired result however it seems to me like this is not the standard way of doing this, do you mind taking a look and improve this solution pls.
Code:
dt1=`date -d 'Jan 19 21:15:00' +%s`
#echo $dt1
while read line
do
	tmstmp=$(echo $line | cut -c '1-15' )
	#echo $tmstmp
	dt2=`date -d "$tmstmp" +%s`
	#echo  $dt2
	echo "Ref time in epoch is $dt1"
	echo "Timestamp from file is $tmstmp"
	echo "Timestamp from file in epoch is $dt2"
	if [[ $dt2 -le $dt1 ]]
	then
		echo "This event occured before 21:15, inserting into new log file"
		echo $line >> filter.txt
	else
		continue 
		
	fi		
	
done < auth.log

# 6  
Old 01-29-2020
That looks like a good approach.

You could get some more efficiency by not calling the date external on each input line and using bash to convert the date time to an mmddHHMMSS number for direct comparison like this:

Code:
#!/bin/bash
function convdtime {
    w="JanFebMarAprMayJunJulAugSepOctNovDec"
    t=${w%%$1*}
    printf -v when "%d%02d%02d%02d%02d\n"  $((${#t}/3 + 1)) $2 ${3//:/ }
}

convdtime Jan 19 21:15:00
start=$when

while read mth day time rest
do
   convdtime $mth $day $time
   if [[ $when -le $start ]]
   then
       echo "This event occured before 21:15, inserting into new log file"
       echo "$mth $day $time $rest" >> filter.txt
   else
      break
   fi
done < auth.log

If file is sorted you can save processing additional lines by using break instead of continue once you are past the target datetime.

Edit: This will have issues around end of calendar year where Dec 31 will appear to be greater than Jan 1

Last edited by Chubler_XL; 01-29-2020 at 05:39 PM..
This User Gave Thanks to Chubler_XL For This Post:
# 7  
Old 01-29-2020
For the fun of it; the (admitted lengthy) "command substitution" finds the last line prior to the date/time given in DT1 on which sed needs to quit:
Code:
DT1="Jan 19 21:10:00"
sed "/$(echo "$DT1" | cut -d" " -f1-3 - file1 | date -f- +"%F %T" | { read T1; { cat; echo $T1; } | sort | grep -B1 "$T1" | head -1 | date -f- +"%b %d %H:%M:%S"; } )/q" file
Jan 19 20:51:48 mukul-Vostro-14-3468 systemd: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Jan 19 20:54:48 mukul-Vostro-14-3468 su[806]: pam_unix(su:session): session closed for user root
Jan 19 20:57:48 mukul-Vostro-14-3468 systemd-logind[1048]: Removed session c49.
Jan 19 20:59:48 mukul-Vostro-14-3468 su[854]: Successful su for root by root
Jan 19 20:57:48 mukul-Vostro-14-3468 su[854]: + ??? root:root
Jan 19 21:05:48 mukul-Vostro-14-3468 su[854]: pam_unix(su:session): session opened for user root by (uid=0)

This User Gave Thanks to RudiC For This Post:

Previous Thread | Next Thread
Test Your Knowledge in Computers #162
Difficulty: Easy
Early versions of MINIX were created by Andrew S. Tanenbaum for commercial purposes.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Awk/sed/cut to filter out records from a file based on criteria

I have two files and would need to filter out records based on certain criteria, these column are of variable lengths, but the lengths are uniform throughout all the records of the file. I have shown a sample of three records below. Line 1-9 is the item number "0227546_1" in the case of the first... (15 Replies)
Discussion started by: MIA651
15 Replies

2. Shell Programming and Scripting

Shell script to filter records in a zip file that contains matching columns from another file

Not sure if this is the correct forum for this question. I have two files. file1.zip, file2 Input: file1.zip col1, col2 , col3 a , b , 0:0:0:0:0:c436:9346:d40b x, y, 0:0:0:0:0:880:39f9:c9a7 m, n , 0:0:0:0:0:80c7:9161:fe00 file2.txt col1 c4:36:93:46:d4:0b... (1 Reply)
Discussion started by: anil.v
1 Replies

3. UNIX for Dummies Questions & Answers

Filter records in a huge text file from a filter text file

Hi Folks, I have a text file with lots of rows with duplicates in the first column, i want to filter out records based on filter columns in a different filter text file. bash scripting is what i need. Data.txt Name OrderID Quantity Sam 123 300 Jay 342 498 Kev 78 2500 Sam 420 50 Vic 10... (3 Replies)
Discussion started by: tech_frk
3 Replies

4. Shell Programming and Scripting

Filter records based on 2nd file

Hello, I want to filter records of a file if they fall in range associated with a second file. First the chr number (2nd col of 1st file and 1st col of 2nd file) needs to be matched. Then if the 3rd col of the first file falls within any of the ranges specified by the 2nd and 3rd cols , then... (4 Replies)
Discussion started by: ritakadm
4 Replies

5. Shell Programming and Scripting

Need to get all the records from a log file greater than timestamp supplied.

I have a log file which has records with hung thread information/error I need to find out hung thread from log file greater than timestamp supplied. 00000026 ThreadMonitor W WSVR0605W: Thread "WebContainer : 1" (00000027) has been active for 701879 milliseconds and may be hung. There is/are... (6 Replies)
Discussion started by: megh
6 Replies

6. Shell Programming and Scripting

Delete log file entries based on the Date/Timestamp within log file

If a log file is in the following format 28-Jul-10 ::: Log message 28-Jul-10 ::: Log message 29-Jul-10 ::: Log message 30-Jul-10 ::: Log message 31-Jul-10 ::: Log message 31-Jul-10 ::: Log message 1-Aug-10 ::: Log message 1-Aug-10 ::: Log message 2-Aug-10 ::: Log message 2-Aug-10 :::... (3 Replies)
Discussion started by: vikram3.r
3 Replies

7. Shell Programming and Scripting

Filter records in a file using AWK

I want to filter records in one of my file using AWK command (or anyother command). I am using the below code awk -F@ '$1=="0003"&&"$2==20100402" print {$0}' $INPUT > $OUTPUT I want to pass the 0003 and 20100402 values through a variable. How can I do this? Any help is much... (1 Reply)
Discussion started by: gpaulose
1 Replies

8. Shell Programming and Scripting

filter out all the records which are having space in the 8th filed of my file

I have a file which is having fileds separtaed by delimiter. Ex: C;4498;qwa;cghy;;;;40;;222122 C;4498;sample;city;;;;34 2;;222123 C;4498;qwe;xcbv;;;;34-2;;222124 C;4498;jj;sffz;;;;41;;222120 C;4498;eert;qwq;;;;34 A;;222125 C;4498;jj;szxzzd;;;;34;;222127 out of these records I... (3 Replies)
Discussion started by: indusri
3 Replies

9. Shell Programming and Scripting

List all log records logged after $timestamp ?

I am trying to find a way to list every records inside a file (usually a log file) that are present after a record mathing/greater-then a timestamp supplied by another script. The timestamp can be anywhere inside the record and it is usually in the standard `date` format (will not look for other... (5 Replies)
Discussion started by: Browser_ice
5 Replies

10. UNIX for Advanced & Expert Users

Copy lines from a log file based on timestamp

how to copy lines from a log file based on timestamp. INFO (RbrProcessFlifoEventSessionEJB.java:processFlight:274) - E_20080521_110754_967: rbrAciInfoObjects listing complete! INFO (RbrPnrProcessEventSessionEJB.java:processFlight:197) - Event Seq: 1647575217; Carrier: UA; Flt#: 0106; Origin:... (1 Reply)
Discussion started by: ranjiadmin
1 Replies

Featured Tech Videos