Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Filter records from a log file based on timestamp Post 303043487 by Chubler_XL on Wednesday 29th of January 2020 03:32:57 PM
Old 01-29-2020
That looks like a good approach.

You could get some more efficiency by not calling the date external on each input line and using bash to convert the date time to an mmddHHMMSS number for direct comparison like this:

Code:
#!/bin/bash
function convdtime {
    w="JanFebMarAprMayJunJulAugSepOctNovDec"
    t=${w%%$1*}
    printf -v when "%d%02d%02d%02d%02d\n"  $((${#t}/3 + 1)) $2 ${3//:/ }
}

convdtime Jan 19 21:15:00
start=$when

while read mth day time rest
do
   convdtime $mth $day $time
   if [[ $when -le $start ]]
   then
       echo "This event occured before 21:15, inserting into new log file"
       echo "$mth $day $time $rest" >> filter.txt
   else
      break
   fi
done < auth.log

If file is sorted you can save processing additional lines by using break instead of continue once you are past the target datetime.

Edit: This will have issues around end of calendar year where Dec 31 will appear to be greater than Jan 1

Last edited by Chubler_XL; 01-29-2020 at 04:39 PM..
This User Gave Thanks to Chubler_XL For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Copy lines from a log file based on timestamp

how to copy lines from a log file based on timestamp. INFO (RbrProcessFlifoEventSessionEJB.java:processFlight:274) - E_20080521_110754_967: rbrAciInfoObjects listing complete! INFO (RbrPnrProcessEventSessionEJB.java:processFlight:197) - Event Seq: 1647575217; Carrier: UA; Flt#: 0106; Origin:... (1 Reply)
Discussion started by: ranjiadmin
1 Replies

2. Shell Programming and Scripting

List all log records logged after $timestamp ?

I am trying to find a way to list every records inside a file (usually a log file) that are present after a record mathing/greater-then a timestamp supplied by another script. The timestamp can be anywhere inside the record and it is usually in the standard `date` format (will not look for other... (5 Replies)
Discussion started by: Browser_ice
5 Replies

3. Shell Programming and Scripting

filter out all the records which are having space in the 8th filed of my file

I have a file which is having fileds separtaed by delimiter. Ex: C;4498;qwa;cghy;;;;40;;222122 C;4498;sample;city;;;;34 2;;222123 C;4498;qwe;xcbv;;;;34-2;;222124 C;4498;jj;sffz;;;;41;;222120 C;4498;eert;qwq;;;;34 A;;222125 C;4498;jj;szxzzd;;;;34;;222127 out of these records I... (3 Replies)
Discussion started by: indusri
3 Replies

4. Shell Programming and Scripting

Filter records in a file using AWK

I want to filter records in one of my file using AWK command (or anyother command). I am using the below code awk -F@ '$1=="0003"&&"$2==20100402" print {$0}' $INPUT > $OUTPUT I want to pass the 0003 and 20100402 values through a variable. How can I do this? Any help is much... (1 Reply)
Discussion started by: gpaulose
1 Replies

5. Shell Programming and Scripting

Delete log file entries based on the Date/Timestamp within log file

If a log file is in the following format 28-Jul-10 ::: Log message 28-Jul-10 ::: Log message 29-Jul-10 ::: Log message 30-Jul-10 ::: Log message 31-Jul-10 ::: Log message 31-Jul-10 ::: Log message 1-Aug-10 ::: Log message 1-Aug-10 ::: Log message 2-Aug-10 ::: Log message 2-Aug-10 :::... (3 Replies)
Discussion started by: vikram3.r
3 Replies

6. Shell Programming and Scripting

Need to get all the records from a log file greater than timestamp supplied.

I have a log file which has records with hung thread information/error I need to find out hung thread from log file greater than timestamp supplied. 00000026 ThreadMonitor W WSVR0605W: Thread "WebContainer : 1" (00000027) has been active for 701879 milliseconds and may be hung. There is/are... (6 Replies)
Discussion started by: megh
6 Replies

7. Shell Programming and Scripting

Filter records based on 2nd file

Hello, I want to filter records of a file if they fall in range associated with a second file. First the chr number (2nd col of 1st file and 1st col of 2nd file) needs to be matched. Then if the 3rd col of the first file falls within any of the ranges specified by the 2nd and 3rd cols , then... (4 Replies)
Discussion started by: ritakadm
4 Replies

8. UNIX for Dummies Questions & Answers

Filter records in a huge text file from a filter text file

Hi Folks, I have a text file with lots of rows with duplicates in the first column, i want to filter out records based on filter columns in a different filter text file. bash scripting is what i need. Data.txt Name OrderID Quantity Sam 123 300 Jay 342 498 Kev 78 2500 Sam 420 50 Vic 10... (3 Replies)
Discussion started by: tech_frk
3 Replies

9. Shell Programming and Scripting

Shell script to filter records in a zip file that contains matching columns from another file

Not sure if this is the correct forum for this question. I have two files. file1.zip, file2 Input: file1.zip col1, col2 , col3 a , b , 0:0:0:0:0:c436:9346:d40b x, y, 0:0:0:0:0:880:39f9:c9a7 m, n , 0:0:0:0:0:80c7:9161:fe00 file2.txt col1 c4:36:93:46:d4:0b... (1 Reply)
Discussion started by: anil.v
1 Replies

10. Shell Programming and Scripting

Awk/sed/cut to filter out records from a file based on criteria

I have two files and would need to filter out records based on certain criteria, these column are of variable lengths, but the lengths are uniform throughout all the records of the file. I have shown a sample of three records below. Line 1-9 is the item number "0227546_1" in the case of the first... (15 Replies)
Discussion started by: MIA651
15 Replies
CAL(1)							    BSD General Commands Manual 						    CAL(1)

NAME
cal -- displays a calendar SYNOPSIS
cal [-3hjry] [-A after] [-B before] [-d day-of-week] [-R reform-spec] [[month] year] DESCRIPTION
cal displays a simple calendar. If arguments are not specified, the current month is displayed. The options are as follows: -3 Same as ``-A 1 -B 1''. -A after Display after months after the specified month. -B before Display before months before the specified month. -d day-of-week Specifies the day of the week on which the calendar should start. Valid values are 0 through 6, presenting Sunday through Saturday, inclusively. The default output starts on Sundays. -h Highlight the current day, if present in the displayed calendar. If output is to a terminal, then the appropriate terminal sequences are used, otherwise overstriking is used. If more than one -h is used and output is to a terminal, the current date will be high- lighted in inverse video instead of bold. -j Display Julian dates (days one-based, numbered from January 1). -R reform-spec Selects an alternate Gregorian reform point from the default of September 3rd, 1752. The reform-spec can be selected by one of the built-in names (see NOTES for a list) or by a date of the form YYYY/MM/DD. The date and month may be omitted, provided that what is specified uniquely selects a given built-in reform point. If an exact date is specified, then that date is taken to be the first missing date of the Gregorian Reform to be applied. -r Display the month in which the Gregorian Reform adjustment was applied, if no other month or year information is given. If used in conjunction with -y, then the entire year is displayed. -y Display a calendar for the current year. If no parameters are specified, the current month's calendar is displayed. A single parameter specifies the year and optionally the month in ISO format: ``cal 2007-12'' Two parameters denote the month (1 - 12) and year. Note that the century must be included in the year. A year starts on Jan 1. NOTES
In the USA and Great Britain the Gregorian Reformation occurred in 1752. By this time, most countries had recognized the reformation (although a few did not recognize it until the 1900's.) Eleven days following September 2, 1752 were eliminated by the reformation, so the calendar for that month is a bit unusual. In view of the chaotic way the Gregorian calendar was adopted throughout the world in the years between 1582 and 1928 make sure to take into account the date of the Gregorian Reformation in your region if you are checking a calendar for a very old date. cal has a decent built-in list of Gregorian Reform dates and the names of the countries where the reform was adopted: Italy Oct. 5, 1582 Denmark Feb. 19, 1700 Spain Oct. 5, 1582 Great Britain Sep. 3, 1752 Portugal Oct. 5, 1582 Sweden Feb. 18, 1753 Poland Oct. 5, 1582 Finland Feb. 18, 1753 France Dec. 12, 1582 Japan Dec. 20, 1872 Luxembourg Dec. 22, 1582 China Nov. 7, 1911 Netherlands Dec. 22, 1582 Bulgaria Apr. 1, 1916 Bavaria Oct. 6, 1583 U.S.S.R. Feb. 1, 1918 Austria Jan. 7, 1584 Serbia Jan. 19, 1919 Switzerland Jan. 12, 1584 Romania Jan. 19, 1919 Hungary Oct. 22, 1587 Greece Mar. 10, 1924 Germany Feb. 19, 1700 Turkey Dec. 19, 1925 Norway Feb. 19, 1700 Egypt Sep. 18, 1928 The country known as Great Britain can also be referred to as England since that has less letters and no spaces in it. This is meant only as a measure of expediency, not as a possible slight to anyone involved. HISTORY
A cal command appeared in Version 6 AT&T UNIX. BSD
December 21, 2007 BSD
All times are GMT -4. The time now is 09:24 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy