Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: How to search backwards in a log file by timestamp of entries?
Top Forums Shell Programming and Scripting How to search backwards in a log file by timestamp of entries? Post 302355759 by kevinmccallum on Wednesday 23rd of September 2009 03:53:28 PM
Old 09-23-2009
Question How to search backwards in a log file by timestamp of entries?
Hello. I'm not nearly good enough with awk/perl to create the logfile scraping script that my boss is insisting we need immediately. Here is a brief 3-line excerpt from the access.log file in question (actual URL domain changed to 'aaa.com'):

Code:
209.253.130.36 - - [23/Sep/2009:12:55:44 -0700] "GET /images/products/en_us/pc/detail/273595_dt.jpg HTTP/1.1" 200 28520 "http://www.aaa.com/product/holiday+parties/halloween+party+supplies.do?" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; FunWebProducts; .NET CLR 1.1.4322)" 22134 "__utma=8470452.136497171.1253643073.1253655989.1253731688.3; __utmb=8470452.4.10.1253731688; __utmz=8470452.1253643073.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); s_cc=true"
99.60.55.157 - - [23/Sep/2009:12:55:45 -0700] "GET /mod/productquickview/includes/themes/default.css HTTP/1.1" 200 767 "http://www.aaa.com/home.do?" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.14) Gecko/2009082707 Firefox/3.0.14 (.NET CLR 3.5.30729)" 14097 "customer=none; basket=none; __utma=8470452.1058319807.1252542208.1252547047.1252713609.3; __utmz=8470452.1252542208.1.1.utmcsr=yahoo|utmccn=(organic)|utmcmd=organic|utmctr=aaa; JSESSIONID=j0d7VJsXNBv6ztnpOp"
198.7.255.226 - - [23/Sep/2009:12:55:46 -0700] "GET /images/products/en_us/gateways/costumes_R_01_C_01.jpg HTTP/1.1" 200 30097 "http://www.aaa.com/category/costumes+%26+accessories.do" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.14) Gecko/2009082707 Firefox/3.0.14 (.NET CLR 3.5.30729)" 12334 "s_cc=true"

So the lines start with an IP-address, followed by date, and then time. We want to only search the last 10 minutes in the file (say if current time is 11:40, we want to only look at lines that go back to 11:30). I've got the code to convert the current time into scalar, subtract 600 secs, and store that time as single character variables (ie: $a = 1, $b = 1, $c = 3, $d = 0).

But I need help with an awk (or other?) code line that will parse each entry in the log file to skip over the IP and the date, and match against the TIMEstamp only. And what's more, we'd like it to do so starting from the bottom of the file (ie: with the most recent entry) and go backwards......and then hopefully stop the search when it hits the first entry that does NOT fall within the past 10-min (because log file is very, very large!).

Any and all help or suggestions would be monumentally appreciated.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Spooling a log file with timestamp

Hi From shell script i am invoking sqlplus to connect to oracle database and then i spool a csv file as with output. What i want to do is to change the file name with timestamp on it so after spooling finish shell script change file name with time stamp. can someone help me to do that . Thanks... (2 Replies)
Discussion started by: ukadmin
2 Replies

2. UNIX for Advanced & Expert Users

Copy lines from a log file based on timestamp

how to copy lines from a log file based on timestamp. INFO (RbrProcessFlifoEventSessionEJB.java:processFlight:274) - E_20080521_110754_967: rbrAciInfoObjects listing complete! INFO (RbrPnrProcessEventSessionEJB.java:processFlight:197) - Event Seq: 1647575217; Carrier: UA; Flt#: 0106; Origin:... (1 Reply)
Discussion started by: ranjiadmin
1 Replies

3. Shell Programming and Scripting

Search backwards

Hi, I have a variable , lets say a=/disk1/net/first.ksh i need to grep "first.ksh" everytime "a" gets changed dynamically and i do not know how many '"/" are there in my variable. Can somebody help me out. (9 Replies)
Discussion started by: giri_luck
9 Replies

4. Shell Programming and Scripting

search backwards relative to a string

Hi, I have to search for first occurenceof string str1 in a file(>5GB). Now, after I have that , I have to search backwards from that offset till I get another string str2. I should also be able to get the new string str2's offset. Similarly, I look for last occurence of str1 and then... (1 Reply)
Discussion started by: finder255
1 Replies

5. Shell Programming and Scripting

concatenate log file lines up to timestamp

Hi, Using sed awk or perl I am trying to do something similar to https://www.unix.com/shell-programming-scripting/105887-sed-awk-concatenate-lines-until-blank-line-2.html but my requirement is slightly different. What I am trying to accomplish is to reformat a logfile such that all lines... (4 Replies)
Discussion started by: AlanC
4 Replies

6. Shell Programming and Scripting

search for string and replace backwards

I'm new to Unix scripting and I'm not sure if this can be done. Example: search (grep) in a file for 'Control ID' and then replace with 4 blanks 7 bytes before 'Control ID. input "xxxxxx1234xxxxxxxControl IDxxxxxx" output: "xxxxxx xxxxxxxControl IDxxxxxx" thanks! (7 Replies)
Discussion started by: jbt828
7 Replies

7. Shell Programming and Scripting

Delete log file entries based on the Date/Timestamp within log file

If a log file is in the following format 28-Jul-10 ::: Log message 28-Jul-10 ::: Log message 29-Jul-10 ::: Log message 30-Jul-10 ::: Log message 31-Jul-10 ::: Log message 31-Jul-10 ::: Log message 1-Aug-10 ::: Log message 1-Aug-10 ::: Log message 2-Aug-10 ::: Log message 2-Aug-10 :::... (3 Replies)
Discussion started by: vikram3.r
3 Replies

8. Programming

How to search a file based on a time stamp backwards 10 seconds

Hi all, I'm after some help with this small issue which i'm struggling to work out a fix for. I have a file that contains records that all have a time stamp for each individual record, i need to search the file for a specific time stamp and then search back 10 seconds to see if the number... (2 Replies)
Discussion started by: sp3arsy
2 Replies

9. Shell Programming and Scripting

Search backwards to certain string

Hi, I'm using the following to do a backwards search of a file for a string sed s/^M//g FILE | nawk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r;print;c=a}b{r=$0}' b=10 a=0 s="9005"|grep "policy "|sort -u |awk '{print $4}'|cut -c2-10 My issue is that because I'm looking back 10 lines it's... (11 Replies)
Discussion started by: SaltyDog
11 Replies

10. Shell Programming and Scripting

Search for latest Timestamp in file

Hi, I have a file which generate Timestamp in this format :- 20121012162354 20121114191610 20121210232808 20121216220002 20130106220002 20130127220001 20130203220001 20121012162354 20121114191610 20121210232808 20121216220002 20130106220002 20130127220001 20130203220001 (2 Replies)
Discussion started by: netdbaind
2 Replies

LEARN ABOUT DEBIAN

set_color

set_color(1)							       fish							      set_color(1)

NAME

       set_color - set_color - set the terminal color

set_color - set the terminal color
   Synopsis
       set_color [-v --version] [-h --help] [-b --background COLOR] [COLOR]

   Description
       Change the foreground and/or background color of the terminal. COLOR is one of black, red, green, brown, yellow, blue, magenta, purple,
       cyan, white and normal.

       o -b, --background Set the background color

       o -c, --print-colors Prints a list of all valid color names

       o -h, --help Display help message and exit

       o -o, --bold Set bold or extra bright mode

       o -u, --underline Set underlined mode

       o -v, --version Display version and exit

       Calling set_color normal will set the terminal color to whatever is the default color of the terminal.

       Some terminals use the --bold escape sequence to switch to a brighter color set. On such terminals, set_color white will result in a grey
       font color, while set_color --bold white will result in a white font color.

       Not all terminal emulators support all these features. This is not a bug in set_color but a missing feature in the terminal emulator.

       set_color uses the terminfo database to look up how to change terminal colors on whatever terminal is in use. Some systems have old and
       incomplete terminfo databases, and may lack color information for terminals that support it. Download and install the latest version of
       ncurses and recompile fish against it in order to fix this issue.

Version 1.23.1							  Sun Jan 8 2012						      set_color(1)
All times are GMT -4. The time now is 01:03 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy