Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Get the line from the log file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Get the line from the log file
# 8  
Old 05-07-2014
Quote:
Originally Posted by Don Cragun
but you have a recent 1993 version of the Korn shell (such as the one on the last few releases of Mac OS X), you could use:
Code:
	$(printf '%(%Y-%b-%d %T)T' '2 hours ago')

Thanks for the info Don. Is it documented somewhere?
I checked ksh93 manual and found :

Quote:
Code:
printf format  [  arg  . . .  ]
.
.
. 
%(date-format)T

A %(date-format)T format can be use to treat an argument as a date/time string and to format the date/time according to the date-format as defined for the date(1) command.


However, I couldn't find if we can use GNU style text for date arithmetic (e.g. 2 hours ago)
Last edited by Don Cragun; 05-07-2014 at 04:58 AM.. Reason: Fix tags.
# 9  
Old 05-07-2014
Hi clx,
I agree that the documentation on the ksh printf %T format specifier is subpar. Through trial and error, I have found that the argument specifying the date and time can be formatted at least using the following formats:
  1. Output from date +%c with or without day-of-week, timezone, or year.
  2. Day-of-week
  3. last day-of-week
  4. next day-of-week
  5. n unit ago (where n is a positive integer value and unit is year, month, week, day, hour, minute, or second or the plural form of any of these). However, month and minute don't behave the way I expect them to.
  6. n unit ahead
You might also notice that the GNU date utility man page doesn't say much about the format of the arguments for its -d option either.
This User Gave Thanks to Don Cragun For This Post:
clx
# 10  
Old 05-07-2014
Hi Don Cragun ,

Thanks much for your responses.Smilie
Here is the details
uname -a 's output

Code:
Linux  SMP Mon Jul 1 17:58:32 EDT 2013 64 GNU/Linux

date command gives the below output.
Code:
Wed May  7 03:49:57 CDT 2014

the script am using is the below one
Code:
#!/bin/ksh
# Initialize start date (sd), start time (st), and end date (ed)
# Note that if this script can be run close to midnight, sd and st msut be set
# before ed.
#EST5EDT
read sd st <<-EOF
	$(TZ=CST8CDT date '+%Y-%b-%d %T')
EOF
ed=$(date '+%Y-%b-%d')
printf "Start date=%s, Start time=%s, End date=%s\n" $sd $st $ed

awk -v ed=$ed -v sd=$sd -v st=$st '
$1 ~ /^[0-9]{4}-[[:alpha:]]{3}-[0-3][0-9]$/ {
        # If the start date matches and the time in this line is later than the
        # start time, print this and later lines.  If sd and st are late
        # yesterday (after 10pm), also print lines from today.
	if(($1 == sd && $2 < st) || ($1 == ed && ed != sd))
		p = 1
	else	p = 0
}
p' log

when i try to run the script i ma getting the below output.
Code:
Start date=2014-May-07, Start time=01:50:44, End date=2014-May-07

sample input log

Code:
2014-May-06 23:57:30 676;WARN   ;RMI TCP Connection(13158);
2014-May-06 23:57:32 688;WARN   ;RMI TCP Connection(13158);
2014-May-07 01:57:32 689;WARN   ;RMI TCP Connection(13158);

Please correct me.

---------- Post updated at 06:07 AM ---------- Previous update was at 04:54 AM ----------

Hi Don,

I undestand the script and now i am able to run the script and getting the output.

One step ahead from this can i able to print the lines in between the timestamp, eventhough it does not start with timestamp.
Something like below,
Code:
2014-May-07 04:00:32 689;WARN   ;RMI TCP Connection(13158)
</Stack>
    </Error>
    <Error ErrorCode="java.sql.SQLRecoverableException"
        ErrorDescription="Error_description_not_available" ErrorRelatedMoreInfo="">
        <Attribute Name="ErrorCode" Value="java.sql.SQLRecoverableException"/>
        <Attribute Name="ErrorDescription" Value="Error_description_not_available"/>
        <Stack>com.yantra.yfc.util.YFCException
2014-May-07 04:10:32 689;WARN   ;RMI TCP Connection(13158)-10.68.154.33;                    ;Clearing cache. Number cached=0

# 11  
Old 05-07-2014
That is what the script does. When it sees a line starting with a date, it decides whether to print that line and the lines following it until it sees another line starting with a date.

What didn't you understand before? Why weren't you getting any output from the awk portion of the script?
# 12  
Old 05-12-2014
Hi All,

Don thanks for your valuable inputs.

I am having the same requirement like mentioned in this thread.i.e to take the lines which are modified within 5 hrs.
Only thing is the timestamp is getting differed like below.

Code:
[5/11/14 6:38:40:748 CDT] 0000001a WSChannelFram A   The Transport Channel Service has started chain 
[5/11/14 6:38:40:836 CDT] 0000001b  The system discovered process (name: nodeagent, type
[5/11/14 6:43:40:598 CDT] 0000001d FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident  emitted on 7456371539064352.txt com.ibm.ws.wsgroup.bb.BBPostingMsg 65
[5/11/14 9:19:02:321 CDT] 00000026 SystemOut     O File Name: /customer_overrides.properties
[5/11/14 9:19:02:324 CDT] 00000026 SystemOut     O File Name: /customer_overrides.properties
[5/11/14 10:40:21:640 CDT] 00000026 SystemOut     O File Name: /customer_overrides.properties

So i modified the script like below,

Code:
sd=$(TZ=CST11CDT date '+%_m/%e/%y' | tr -d ' ')
st=$(TZ=CST11CDT date '+%H:%M:%S')
ed=$(date '+%_m/%e/%y'| tr -d ' ')

Code:
cut -c 2- | sed 's/:/ /3'awk -v ed=$ed -v sd=$sd -v st="21:49:52" '
{ if(($1 == sd && $2 < st) || ($1 == ed && ed != sd)) p=1; }p' test.txt

Please help me on this as am not getting proper output
# 13  
Old 05-12-2014
Quote:
Originally Posted by mohanalakshmi
Hi All,

Don thanks for your valuable inputs.

I am having the same requirement like mentioned in this thread.i.e to take the lines which are modified within 5 hrs.
Only thing is the timestamp is getting differed like below.

Code:
[5/11/14 6:38:40:748 CDT] 0000001a WSChannelFram A   The Transport Channel Service has started chain 
[5/11/14 6:38:40:836 CDT] 0000001b  The system discovered process (name: nodeagent, type
[5/11/14 6:43:40:598 CDT] 0000001d FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident  emitted on 7456371539064352.txt com.ibm.ws.wsgroup.bb.BBPostingMsg 65
[5/11/14 9:19:02:321 CDT] 00000026 SystemOut     O File Name: /customer_overrides.properties
[5/11/14 9:19:02:324 CDT] 00000026 SystemOut     O File Name: /customer_overrides.properties
[5/11/14 10:40:21:640 CDT] 00000026 SystemOut     O File Name: /customer_overrides.properties

So i modified the script like below,

Code:
sd=$(TZ=CST11CDT date '+%_m/%e/%y' | tr -d ' ')
st=$(TZ=CST11CDT date '+%H:%M:%S')
ed=$(date '+%_m/%e/%y'| tr -d ' ')

Code:
cut -c 2- | sed 's/:/ /3'awk -v ed=$ed -v sd=$sd -v st="21:49:52" '
{ if(($1 == sd && $2 < st) || ($1 == ed && ed != sd)) p=1; }p' test.txt

Please help me on this as am not getting proper output
There are several problems here:
  1. If there is ANY chance that your script could be run close to midnight, NEVER set the starting date and timestamp in separate calls to the date utility.
  2. Using tr to get rid of spaces at the start of the month and day fields in dates is perfectly reasonable, but relatively expensive. Using shell built-ins to handle this issue is much more efficient.
  3. The input to the cut command in this pipeline is standard input for the script (not your log file).
  4. Adding a leading "[" to the date strings is much more efficient than invoking a separate process to run cut to remove the 1st character from each line in your log files.
  5. sed 's/:/ /3'awk is not a valid sed command. I assume that you intended to have sed 's/:/ /3' | awk instead, but when doing string comparisons to compare timestamps, getting rid of the colon and the milliseconds won't affect the results.
  6. Your use of %H (and my use of %T) present the hour as a two digit value with a leading "0" for times before 10am. Your sample input omits the leading "0" and you didn't make any adjustments to account for that. (Note that a leading "0" needs to be added to your input data (when needed); removing a leading "0" (if present) from ts will not correctly check for the desired times.)
If I correctly understand your new input file format, the following might do what you want:
Code:
#!/bin/ksh
# Initialize start date (sd), start time (st), and end date (ed).
# Note that if this script can be run close to midnight, sd and st msut be set
# before ed.  The read command will strip off leading spaces in any of the
# values and the command substitutions will strip off the trailing newlines
# provided by the date commands.  Spaces in the date format strings guarantee
# that sd1 and ed1 will be set to the 1- or 2-digit month and sd2 and ed2 will
# be set to the 1- or 2-digit day a "/" and the last two digits of the year.
read sd1 sd2 st ed1 ed2 <<-EOF
	$(TZ=CST11CDT date '+%_m %e/%y %T') $(date '+%_m %e/%y')
EOF
# Add leading "[" and separating "/" to ed and sd.
ed="[$ed1/$ed2"
sd="[$sd1/$sd2"
printf "Start date=%s, Start time=%s, End date=%s\n" $sd $st $ed

awk -v ed="$ed" -v sd="$sd" -v st="$st" '
{	# Normalize timestamp field to have a 2-digit leading zero-filled hour
	# before the first colon.
	t = substr($2, 2, 1) == ":" ? "0"$2 : $2
	if(($1 == sd && t > st) || ($1 == ed && ed != sd)) print
}' logfile2

# 14  
Old 10-09-2014
HI i have a doubt on the below command,
Code:
TZ=CST7CDT date '+%Y-%b-%d %T'

This gives the time which is 1 hr before time from the current time.
May i know how to get the time which is past 15 mins .
i.e if current time is
Code:
2014-Oct-08 22:30:51

my expected output is
Code:
2014-Oct-08 22:15:51

Thanks for your help.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Zabbix item for last line of a log file

Dear all,Zabbix version : 2.4 (yes, I know, upgrading soon - honest) Server OS version : CentOS 6, 64-bit (CentOS 7 with the Zabbix upgrade)I've got a large log file that I would like to read by an external process. It's basically the same as reading the item value on a web-page. I have... (5 Replies)
Discussion started by: rbatte1
5 Replies

2. Shell Programming and Scripting

Reading line by line from live log file using while loop and considering only those lines start from

Hi, I want to read a live log file line by line and considering those line which start from time stamp; Below code I am using, which read line but throws an exception when comparing line that does not contain error code tail -F /logs/COMMON-ERROR.log | while read myline; do... (2 Replies)
Discussion started by: ketanraut
2 Replies

3. Shell Programming and Scripting

HELP: Shell Script to read a Log file line by line and extract Info based on KEYWORDS matching

I have a LOG file which looks like this Import started at: Mon Jul 23 02:13:01 EDT 2012 Initialization completed in 2.146 seconds. -------------------------------------------------------------------------------- -- Import summary for Import item: PolicyInformation... (8 Replies)
Discussion started by: biztank
8 Replies

4. Shell Programming and Scripting

How to process log file line by line?

Greetings, I'm new to this forum, also new to shell script I have done some simple shell script before, like backup linux machine using rsync and crontab, but now I need to do some log analyzing, which is beyond my ability... so I'm going to seek for help in this forum, hope someone could give... (5 Replies)
Discussion started by: lunaticdawn
5 Replies

5. Shell Programming and Scripting

parse a log file and remember last line

Hi all: I'm working on a HPUX 11.23 system and I am needing to parse a tomcat-jakarta log file for memory use. Getting the desired data is easy, assuming the log file does not grow. This file grows constantly and I want to check it q 5 min. The next check will pick up from where it left off 5... (4 Replies)
Discussion started by: raggmopp
4 Replies

6. Shell Programming and Scripting

How to find duplicate line in log file?

Hi guys, I'm really happy to find this forum I have a log file, and I have to find all lines that have "error" word, and then save this output in file, the output file has to have just only one line to any Duplicated lines and counter that show how many time this lines duplicated? I already... (2 Replies)
Discussion started by: wax_light
2 Replies

7. Shell Programming and Scripting

shell script to read a line in gps receiver log file and append that line to new file

Hi, I have gps receiver log..its giving readings .like below Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GPSD,R=1 $GPGSV,3,1,11,08,16,328,40,11,36,127,00,28,33,283,39,20,11,165,00*71... (3 Replies)
Discussion started by: gudivada213
3 Replies

8. Shell Programming and Scripting

How to read multiple line from log file

I have errors in the log that span multiple lines and I can find say the 2nd line in the log of this error using an unique word. However, this only gets me the line that the word appears in not the full error which may be 3 or four line long. So if there way to display say the line before a match... (4 Replies)
Discussion started by: vishal_vsh1
4 Replies

9. Shell Programming and Scripting

i want to add one new line in log file

Help There are so many lines in log file like 'SQL> spool off' just I want to add one new line after this to seperate each one eg; SQL> spool off ------------------------------- SQL> spool off ------------------------------- SQL> spool off ------------------------------- (2 Replies)
Discussion started by: suryanarayana
2 Replies

10. Shell Programming and Scripting

checking size of the first line in a log file

Hi My test.log file looks like this: 0 190_GSTV_HUX_003QISCGSK026_error070322_115331917.log 34 190_GSTV_HUX_003QISCGSK026_error070117_151311385.log 12 190_GSTV_HUX_003QISCGSK026_error070117_151230001.log 2 190_GSTV_HUX_003QISCGSK026_error070117_101010001.log 0... (19 Replies)
Discussion started by: kiran1112
19 Replies
Login or Register to Ask a Question