Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Get the line from the log file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Get the line from the log file
# 1  
Old 05-06-2014
Get the line from the log file
Hi All,

My requirement is to get the lines from the logfile which should updated in the last two hrs from the current time.
Code:
2014-May-05 03:24:07 525;WARN   ;error in line 1
at (Unknown Source)
	at mpl.java:37)
	at (Method.java:611)
	at (ApiHelper.java:395)
2014-May-05 03:24:09 780;WARN   ;system error in line 1
at (Unknown Source)
	at mpl.java:37)
	at (Method.java:611)
	at (ApiHelper.java:395)
2014-May-06 03:24:09 781;WARN   ;system error occuerd.
at (Unknown Source)
	at mpl.java:37)
	at (Method.java:611)
	at (ApiHelper.java:395)
2014-May-06 04:24:09 781;WARN   ;system error occuerd.
at (Unknown Source)
	at mpl.java:37)
	at (Method.java:611)
	at (ApiHelper.java:395)

if above is the sample file, current time is May'2014 5.00 AM then i want the lines which is having timestamp of last two hrs from the current time.
Please help me on this.
# 2  
Old 05-06-2014
Date math is hard. If you have GNU date, then you can use this

Code:
mute@thedoctor:~$ ./script < input
2014-May-06 03:24:09 781;WARN   ;system error occuerd.
at (Unknown Source)
        at mpl.java:37)
        at (Method.java:611)
        at (ApiHelper.java:395)
2014-May-06 04:24:09 781;WARN   ;system error occuerd.
at (Unknown Source)
        at mpl.java:37)
        at (Method.java:611)
        at (ApiHelper.java:395)


Code:
mute@thedoctor:~$ cat script
#!/usr/bin/awk -f

BEGIN {
#       was used for testing with specific date/time
        now="May 6 2014 05:00:00"
        cmd=sprintf("date -d'%s 2 hours ago' +%%s", now)
#       cmd="date +%s"
        cmd | getline start
        close(cmd)
}

# if we found a line with a date within two hours,
# then the rest must be in range too
go==1 { print; next }

# reading a date line
$1 ~ /^[0-9][0-9][0-9][0-9]-/ {
        # reformat date from "2014-May-5" to "May 5 2014"
        split($1,a,/-/)
        cmd=sprintf("date -d'%s %s %s %s' +%%s", a[2], a[3], a[1], $2)
        cmd | getline secs
        close(cmd)
        if (secs >= start) {
                print
                go=1
        }
}

This User Gave Thanks to neutronscott For This Post:
mohanalakshmi
# 3  
Old 05-07-2014
If you don't have access to the GNU utilities version of the date utility, but you have a recent 1993 version of the Korn shell (such as the one on the last few releases of Mac OS X), you could use:
Code:
#!/bin/ksh
# Initialize start date (sd), start time (st), and end date (ed)
# Note that if this script can be run close to midnight, sd and st msut be set
# before ed.
read sd st <<-EOF
	$(printf '%(%Y-%b-%d %T)T' '2 hours ago')
EOF
ed=$(date '+%Y-%b-%d')
printf "Start date=%s, Start time=%s, End date=%s\n" $sd $st $ed

awk -v ed=$ed -v sd=$sd -v st=$st '
$1 ~ /^[0-9]{4}-[[:alpha:]]{3}-[0-3][0-9]$/ {
        # If the start date matches and the time in this line is later than the
        # start time, print this and later lines.  If sd and st are late
        # yesterday (after 10pm), also print lines from today.
	if(($1 == sd && $2 > st) || ($1 == ed && ed != sd))
		p = 1
	else	p = 0
}
p' logfile

If you don't have a recent version of ksh93 for start times shifted a few hours from the current time (like the 2 hours needed for this problem), you can use the POSIX method of specifying the time zone to shift the time, but the value you need to use depend on your current timezone.

For example, I am in the US Pacific timezone which can be specified by setting TZ=PST8PDT. To shift the output of date to report times 2 hours ago, add two to the number in TZ for your timezone (e.g., TZ=PST10PDT). To be sure that you have the right value, verify that the command:
Code:
TZ=PST10PDT date '+%Y-%b-%d %T\n'

(with your setting for TZ) prints the date and time two hours ago. Then you can change:
Code:
	$(printf '%(%Y-%b-%d %T)T\n' '2 hours ago')

in the above script to:
Code:
	$(TZ=PST10PDT date '+%Y-%b-%d %T')

(with your setting for TZ) and the script should work with any version of the Korn shell or any other shell (such as bash) that recognizes basic POSIX shell syntax.

If you want to try this on a Solaris/SunOS system, also change awk to /usr/xpg4/bin/awk or /usr/xpg6/bin/awk.
These 2 Users Gave Thanks to Don Cragun For This Post:
clx mohanalakshmi
# 4  
Old 05-07-2014
Hi All,

Thanks for your replies Smilie

I used Don Cragun's reply and i am just getting the below output
Code:
Start date=2014-May-06, Start time=22:15:30, End date=2014-May-07

not the lines from the logfile.Any things need to be added.Please help me.
# 5  
Old 05-07-2014
Your sample input no longer has any data with a date and time that occurs in the last two hours.
# 6  
Old 05-07-2014
Hi ,

Thanks much for your response.Smiliethis is the sample data which i have ,

Code:
2014-May-06 23:17:35 347;WARN   ;RMI TCP Connection(13147)
2014-May-06 23:17:35 347;WARN   ;RMI TCP Connection(13147)
2014-May-06 23:36:00 612;WARN   ;RMI TCP Connection(13154)
2014-May-06 23:57:30 676;WARN   ;RMI TCP Connection(13158)
2014-May-06 23:57:30 676;WARN   ;RMI TCP Connection(13158)
2014-May-06 23:57:32 688;WARN   ;RMI TCP Connection(13158)
2014-May-07 00:57:32 689;WARN   ;RMI TCP Connection(13158)

ouptnow
Code:
Start date=2014-May-06, Start time=22:50:40, End date=2014-May-07

please correct me
# 7  
Old 05-07-2014
Please show us the output from the commands:
Code:
uname -a
date

and, since I suggested several possible variations on the script; show us the exact script that you used, tell us what shell you used, and show us the exact command you used to invoke the script.

As I said before, I'm in the US Pacific timezone (where the output from date is now Tue May 6 23:32:04 PDT 2014). With your latest sample data, I'm getting:
Code:
Start date=2014-May-06, Start time=21:32:04, End date=2014-May-06
2014-May-06 23:17:35 347;WARN   ;RMI TCP Connection(13147)
2014-May-06 23:17:35 347;WARN   ;RMI TCP Connection(13147)
2014-May-06 23:36:00 612;WARN   ;RMI TCP Connection(13154)
2014-May-06 23:57:30 676;WARN   ;RMI TCP Connection(13158)
2014-May-06 23:57:30 676;WARN   ;RMI TCP Connection(13158)
2014-May-06 23:57:32 688;WARN   ;RMI TCP Connection(13158)

If I run it again in a half an hour (when the date here is May 7th), I'll get the above lines and the line:
Code:
2014-May-07 00:57:32 689;WARN   ;RMI TCP Connection(13158)

It is, however, interesting that you have data in your log file that should not have been written until seven minutes after you ran the script. (You showed a start time of 22:50:40 on May 6, so you ran the program at 00:50:40 on May 7???)
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Zabbix item for last line of a log file

Dear all,Zabbix version : 2.4 (yes, I know, upgrading soon - honest) Server OS version : CentOS 6, 64-bit (CentOS 7 with the Zabbix upgrade)I've got a large log file that I would like to read by an external process. It's basically the same as reading the item value on a web-page. I have... (5 Replies)
Discussion started by: rbatte1
5 Replies

2. Shell Programming and Scripting

Reading line by line from live log file using while loop and considering only those lines start from

Hi, I want to read a live log file line by line and considering those line which start from time stamp; Below code I am using, which read line but throws an exception when comparing line that does not contain error code tail -F /logs/COMMON-ERROR.log | while read myline; do... (2 Replies)
Discussion started by: ketanraut
2 Replies

3. Shell Programming and Scripting

HELP: Shell Script to read a Log file line by line and extract Info based on KEYWORDS matching

I have a LOG file which looks like this Import started at: Mon Jul 23 02:13:01 EDT 2012 Initialization completed in 2.146 seconds. -------------------------------------------------------------------------------- -- Import summary for Import item: PolicyInformation... (8 Replies)
Discussion started by: biztank
8 Replies

4. Shell Programming and Scripting

How to process log file line by line?

Greetings, I'm new to this forum, also new to shell script I have done some simple shell script before, like backup linux machine using rsync and crontab, but now I need to do some log analyzing, which is beyond my ability... so I'm going to seek for help in this forum, hope someone could give... (5 Replies)
Discussion started by: lunaticdawn
5 Replies

5. Shell Programming and Scripting

parse a log file and remember last line

Hi all: I'm working on a HPUX 11.23 system and I am needing to parse a tomcat-jakarta log file for memory use. Getting the desired data is easy, assuming the log file does not grow. This file grows constantly and I want to check it q 5 min. The next check will pick up from where it left off 5... (4 Replies)
Discussion started by: raggmopp
4 Replies

6. Shell Programming and Scripting

How to find duplicate line in log file?

Hi guys, I'm really happy to find this forum I have a log file, and I have to find all lines that have "error" word, and then save this output in file, the output file has to have just only one line to any Duplicated lines and counter that show how many time this lines duplicated? I already... (2 Replies)
Discussion started by: wax_light
2 Replies

7. Shell Programming and Scripting

shell script to read a line in gps receiver log file and append that line to new file

Hi, I have gps receiver log..its giving readings .like below Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GPSD,R=1 $GPGSV,3,1,11,08,16,328,40,11,36,127,00,28,33,283,39,20,11,165,00*71... (3 Replies)
Discussion started by: gudivada213
3 Replies

8. Shell Programming and Scripting

How to read multiple line from log file

I have errors in the log that span multiple lines and I can find say the 2nd line in the log of this error using an unique word. However, this only gets me the line that the word appears in not the full error which may be 3 or four line long. So if there way to display say the line before a match... (4 Replies)
Discussion started by: vishal_vsh1
4 Replies

9. Shell Programming and Scripting

i want to add one new line in log file

Help There are so many lines in log file like 'SQL> spool off' just I want to add one new line after this to seperate each one eg; SQL> spool off ------------------------------- SQL> spool off ------------------------------- SQL> spool off ------------------------------- (2 Replies)
Discussion started by: suryanarayana
2 Replies

10. Shell Programming and Scripting

checking size of the first line in a log file

Hi My test.log file looks like this: 0 190_GSTV_HUX_003QISCGSK026_error070322_115331917.log 34 190_GSTV_HUX_003QISCGSK026_error070117_151311385.log 12 190_GSTV_HUX_003QISCGSK026_error070117_151230001.log 2 190_GSTV_HUX_003QISCGSK026_error070117_101010001.log 0... (19 Replies)
Discussion started by: kiran1112
19 Replies
Login or Register to Ask a Question