Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Filter records from a log file based on timestamp Post 303043487 by Chubler_XL on Wednesday 29th of January 2020 03:32:57 PM
Old 01-29-2020
That looks like a good approach.

You could get some more efficiency by not calling the date external on each input line and using bash to convert the date time to an mmddHHMMSS number for direct comparison like this:

Code:
#!/bin/bash
function convdtime {
    w="JanFebMarAprMayJunJulAugSepOctNovDec"
    t=${w%%$1*}
    printf -v when "%d%02d%02d%02d%02d\n"  $((${#t}/3 + 1)) $2 ${3//:/ }
}

convdtime Jan 19 21:15:00
start=$when

while read mth day time rest
do
   convdtime $mth $day $time
   if [[ $when -le $start ]]
   then
       echo "This event occured before 21:15, inserting into new log file"
       echo "$mth $day $time $rest" >> filter.txt
   else
      break
   fi
done < auth.log

If file is sorted you can save processing additional lines by using break instead of continue once you are past the target datetime.

Edit: This will have issues around end of calendar year where Dec 31 will appear to be greater than Jan 1

Last edited by Chubler_XL; 01-29-2020 at 04:39 PM..
This User Gave Thanks to Chubler_XL For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Copy lines from a log file based on timestamp

how to copy lines from a log file based on timestamp. INFO (RbrProcessFlifoEventSessionEJB.java:processFlight:274) - E_20080521_110754_967: rbrAciInfoObjects listing complete! INFO (RbrPnrProcessEventSessionEJB.java:processFlight:197) - Event Seq: 1647575217; Carrier: UA; Flt#: 0106; Origin:... (1 Reply)
Discussion started by: ranjiadmin
1 Replies

2. Shell Programming and Scripting

List all log records logged after $timestamp ?

I am trying to find a way to list every records inside a file (usually a log file) that are present after a record mathing/greater-then a timestamp supplied by another script. The timestamp can be anywhere inside the record and it is usually in the standard `date` format (will not look for other... (5 Replies)
Discussion started by: Browser_ice
5 Replies

3. Shell Programming and Scripting

filter out all the records which are having space in the 8th filed of my file

I have a file which is having fileds separtaed by delimiter. Ex: C;4498;qwa;cghy;;;;40;;222122 C;4498;sample;city;;;;34 2;;222123 C;4498;qwe;xcbv;;;;34-2;;222124 C;4498;jj;sffz;;;;41;;222120 C;4498;eert;qwq;;;;34 A;;222125 C;4498;jj;szxzzd;;;;34;;222127 out of these records I... (3 Replies)
Discussion started by: indusri
3 Replies

4. Shell Programming and Scripting

Filter records in a file using AWK

I want to filter records in one of my file using AWK command (or anyother command). I am using the below code awk -F@ '$1=="0003"&&"$2==20100402" print {$0}' $INPUT > $OUTPUT I want to pass the 0003 and 20100402 values through a variable. How can I do this? Any help is much... (1 Reply)
Discussion started by: gpaulose
1 Replies

5. Shell Programming and Scripting

Delete log file entries based on the Date/Timestamp within log file

If a log file is in the following format 28-Jul-10 ::: Log message 28-Jul-10 ::: Log message 29-Jul-10 ::: Log message 30-Jul-10 ::: Log message 31-Jul-10 ::: Log message 31-Jul-10 ::: Log message 1-Aug-10 ::: Log message 1-Aug-10 ::: Log message 2-Aug-10 ::: Log message 2-Aug-10 :::... (3 Replies)
Discussion started by: vikram3.r
3 Replies

6. Shell Programming and Scripting

Need to get all the records from a log file greater than timestamp supplied.

I have a log file which has records with hung thread information/error I need to find out hung thread from log file greater than timestamp supplied. 00000026 ThreadMonitor W WSVR0605W: Thread "WebContainer : 1" (00000027) has been active for 701879 milliseconds and may be hung. There is/are... (6 Replies)
Discussion started by: megh
6 Replies

7. Shell Programming and Scripting

Filter records based on 2nd file

Hello, I want to filter records of a file if they fall in range associated with a second file. First the chr number (2nd col of 1st file and 1st col of 2nd file) needs to be matched. Then if the 3rd col of the first file falls within any of the ranges specified by the 2nd and 3rd cols , then... (4 Replies)
Discussion started by: ritakadm
4 Replies

8. UNIX for Dummies Questions & Answers

Filter records in a huge text file from a filter text file

Hi Folks, I have a text file with lots of rows with duplicates in the first column, i want to filter out records based on filter columns in a different filter text file. bash scripting is what i need. Data.txt Name OrderID Quantity Sam 123 300 Jay 342 498 Kev 78 2500 Sam 420 50 Vic 10... (3 Replies)
Discussion started by: tech_frk
3 Replies

9. Shell Programming and Scripting

Shell script to filter records in a zip file that contains matching columns from another file

Not sure if this is the correct forum for this question. I have two files. file1.zip, file2 Input: file1.zip col1, col2 , col3 a , b , 0:0:0:0:0:c436:9346:d40b x, y, 0:0:0:0:0:880:39f9:c9a7 m, n , 0:0:0:0:0:80c7:9161:fe00 file2.txt col1 c4:36:93:46:d4:0b... (1 Reply)
Discussion started by: anil.v
1 Replies

10. Shell Programming and Scripting

Awk/sed/cut to filter out records from a file based on criteria

I have two files and would need to filter out records based on certain criteria, these column are of variable lengths, but the lengths are uniform throughout all the records of the file. I have shown a sample of three records below. Line 1-9 is the item number "0227546_1" in the case of the first... (15 Replies)
Discussion started by: MIA651
15 Replies
MKTIME(3)								 1								 MKTIME(3)

mktime - Get Unix timestamp for a date

SYNOPSIS
int mktime ([int $hour = date("H")], [int $minute = date("i")], [int $second = date("s")], [int $month = date("n")], [int $day = date("j")], [int $year = date("Y")], [int $is_dst = -1]) DESCRIPTION
Returns the Unix timestamp corresponding to the arguments given. This timestamp is a long integer containing the number of seconds between the Unix Epoch (January 1 1970 00:00:00 GMT) and the time specified. Arguments may be left out in order from right to left; any arguments thus omitted will be set to the current value according to the local date and time. NOTES
Note As of PHP 5.1, when called with no arguments, mktime(3) throws an E_STRICT notice: use the time(3) function instead. PARAMETERS
o $hour - The number of the hour relative to the start of the day determined by $month, $day and $year. Negative values reference the hour before midnight of the day in question. Values greater than 23 reference the appropriate hour in the following day(s). o $minute - The number of the minute relative to the start of the $hour. Negative values reference the minute in the previous hour. Values greater than 59 reference the appropriate minute in the following hour(s). o $second - The number of seconds relative to the start of the $minute. Negative values reference the second in the previous minute. Values greater than 59 reference the appropriate second in the following minute(s). o $month - The number of the month relative to the end of the previous year. Values 1 to 12 reference the normal calendar months of the year in question. Values less than 1 (including negative values) reference the months in the previous year in reverse order, so 0 is December, -1 is November, etc. Values greater than 12 reference the appropriate month in the following year(s). o $day - The number of the day relative to the end of the previous month. Values 1 to 28, 29, 30 or 31 (depending upon the month) refer- ence the normal days in the relevant month. Values less than 1 (including negative values) reference the days in the previous month, so 0 is the last day of the previous month, -1 is the day before that, etc. Values greater than the number of days in the relevant month reference the appropriate day in the following month(s). o $year - The number of the year, may be a two or four digit value, with values between 0-69 mapping to 2000-2069 and 70-100 to 1970-2000. On systems where time_t is a 32bit signed integer, as most common today, the valid range for $year is somewhere between 1901 and 2038. However, before PHP 5.1.0 this range was limited from 1970 to 2038 on some systems (e.g. Windows). o $is_dst - This parameter can be set to 1 if the time is during daylight savings time (DST), 0 if it is not, or -1 (the default) if it is unknown whether the time is within daylight savings time or not. If it's unknown, PHP tries to figure it out itself. This can cause unexpected (but not incorrect) results. Some times are invalid if DST is enabled on the system PHP is running on or $is_dst is set to 1. If DST is enabled in e.g. 2:00, all times between 2:00 and 3:00 are invalid and mktime(3) returns an undefined (usu- ally negative) value. Some systems (e.g. Solaris 8) enable DST at midnight so time 0:30 of the day when DST is enabled is evalu- ated as 23:30 of the previous day. Note As of PHP 5.1.0, this parameter became deprecated. As a result, the new timezone handling features should be used instead. Note This parameter has been removed in PHP 7.0.0. RETURN VALUES
mktime(3) returns the Unix timestamp of the arguments given. If the arguments are invalid, the function returns FALSE (before PHP 5.1 it returned -1). ERRORS
/EXCEPTIONS Every call to a date/time function will generate a E_NOTICE if the time zone is not valid, and/or a E_STRICT or E_WARNING message if using the system settings or the $TZ environment variable. See also date_default_timezone_set(3) CHANGELOG
+--------+---------------------------------------------------+ |Version | | | | | | | Description | | | | +--------+---------------------------------------------------+ | 7.0.0 | | | | | | | $is_dst parameter has been removed. | | | | | 5.3.0 | | | | | | | mktime(3) now throws E_DEPRECATED notice if the | | | $is_dst parameter is used. | | | | | 5.1.0 | | | | | | | The $is_dst parameter became deprecated. Made | | | the function return FALSE on error, instead of | | | -1. Fixed the function to accept the year, month | | | and day to be all passed as zero. | | | | | 5.1.0 | | | | | | | When called with no arguments, mktime(3) throws | | | E_STRICT notice. Use the time(3) function | | | instead. | | | | | 5.1.0 | | | | | | | Now issues the E_STRICT and E_NOTICE time zone | | | errors. | | | | +--------+---------------------------------------------------+ EXAMPLES
Example #1 mktime(3) basic example <?php // Set the default timezone to use. Available as of PHP 5.1 date_default_timezone_set('UTC'); // Prints: July 1, 2000 is on a Saturday echo "July 1, 2000 is on a " . date("l", mktime(0, 0, 0, 7, 1, 2000)); // Prints something like: 2006-04-05T01:02:03+00:00 echo date('c', mktime(1, 2, 3, 4, 5, 2006)); ?> Example #2 mktime(3) example mktime(3) is useful for doing date arithmetic and validation, as it will automatically calculate the correct value for out-of-range input. For example, each of the following lines produces the string "Jan-01-1998". <?php echo date("M-d-Y", mktime(0, 0, 0, 12, 32, 1997)); echo date("M-d-Y", mktime(0, 0, 0, 13, 1, 1997)); echo date("M-d-Y", mktime(0, 0, 0, 1, 1, 1998)); echo date("M-d-Y", mktime(0, 0, 0, 1, 1, 98)); ?> Example #3 Last day of a month The last day of any given month can be expressed as the "0" day of the next month, not the -1 day. Both of the following examples will produce the string "The last day in Feb 2000 is: 29". <?php $lastday = mktime(0, 0, 0, 3, 0, 2000); echo strftime("Last day in Feb 2000 is: %d", $lastday); $lastday = mktime(0, 0, 0, 4, -31, 2000); echo strftime("Last day in Feb 2000 is: %d", $lastday); ?> NOTES
Caution Before PHP 5.1.0, negative timestamps were not supported under any known version of Windows and some other systems as well. There- fore the range of valid years was limited to 1970 through 2038. SEE ALSO
checkdate(3), gmmktime(3), date(3), time(3). PHP Documentation Group MKTIME(3)
All times are GMT -4. The time now is 05:31 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy