Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Piping tail to awk to parse a log file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Piping tail to awk to parse a log file
# 1  
Old 05-31-2009
Piping tail to awk to parse a log file
Hello all,

I've got what I'm pretty sure is a simple problem, but I just can't seem to work past it. I'm trying to use awk to pretty up a log file, and calculate a percentage.

The log file looks like this:

Code:
[root@mys-nycnm-srv01 rtaStats]# tail strtovrUsage 
20090531-18:15:45  RSreq - 24, RSsuc - 24, RSrun - 78, RSerr - 0,
20090531-18:25:45  RSreq - 17, RSsuc - 17, RSrun - 72, RSerr - 0,
20090531-18:35:45  RSreq - 19, RSsuc - 19, RSrun - 72, RSerr - 0,
20090531-18:45:45  RSreq - 16, RSsuc - 16, RSrun - 60, RSerr - 0,
20090531-18:55:45  RSreq - 14, RSsuc - 14, RSrun - 62, RSerr - 0,
20090531-19:05:46  RSreq - 27, RSsuc - 26, RSrun - 61, RSerr - 1,
20090531-19:15:46  RSreq - 37, RSsuc - 37, RSrun - 74, RSerr - 0,
20090531-19:25:46  RSreq - 48, RSsuc - 46, RSrun - 84, RSerr - 2,
20090531-19:35:46  RSreq - 54, RSsuc - 54, RSrun - 106, RSerr - 0,
20090531-19:36:52  RSreq - 0, RSsuc - 0, RSrun - 107, RSerr - 0,


If I fun something simple like:

Code:
[root@mys-nycnm-srv01 ~]# cat `which SOperf`
 tail -f /usr/local/mystro/scripts/twctools/rtaStats/strtovrUsage|awk '{print "Requests = " $4 "Success = " $7  "Running Streams = " $10 "Errors = " $13 "Percentage = " ($7 / $4 )*100 }'

[root@mys-nycnm-srv01 ~]# SOperf 
Requests = 47,Success = 47,Running Streams = 122,Errors = 0,Percentage = 100
Requests = 62,Success = 60,Running Streams = 122,Errors = 2,Percentage = 96.7742
Requests = 56,Success = 56,Running Streams = 132,Errors = 0,Percentage = 100
Requests = 50,Success = 50,Running Streams = 124,Errors = 0,Percentage = 100
Requests = 65,Success = 63,Running Streams = 138,Errors = 0,Percentage = 96.9231

It works, but it's a little simpler than what I'd like. I tried doing the following:

Code:
#!/bin/bash

printf "Time\t\t\tRequests\tSuccess\tRunning\tError\tPercentage\n"

tail   /usr/local/mystro/scripts/twctools/rtaStats/strtovrUsage |\
awk         \
'***** = $1} \
 {req = $4} \
 {suc = $7} \
 {err = $13}\
 {run = $10}\
  END       \
{print tim "\t" req "\t\t" suc "\t" run "\t" err "\t" (suc / req)*100 }'

Time                    Requests        Success Running Error   Percentage
20090531-21:35:47       65,             64,     146,    1,      98.4615

But I only get the one line of results, and it won't work with a tail -f.
If it matters, here is the information for the system I'm working on.

Code:
Linux mys-nycnm-srv01 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 i686 i686 i386 GNU/Linux
/usr/bin/awk: symbolic link to `../../bin/gawk'

Also, is there an effective way to work in "divide by 0" error protection into what I'm trying to do (bearing in mind my obviously low skill level?)
# 2  
Old 05-31-2009
Quote:
Originally Posted by DeCoTwc
Also, is there an effective way to work in "divide by 0" error protection into what I'm trying to do (bearing in mind my obviously low skill level?)
check for 0, or use >0
# 3  
Old 06-01-2009
Quote:
Originally Posted by ghostdog74
check for 0, or use >0
Thanks for the advice, shortly after posting I came up with some ideas using some basic if/else for the divide by 0 (though I've yet to try putting it into practice). I'm not sure what you mean by use >0 though?

Either way, my main concern is why the second method of running the script doesn't work, even though the first does. It seems piping tail to awk works, but only sometimes.
# 4  
Old 06-01-2009
Quote:
Originally Posted by DeCoTwc
Thanks for the advice, shortly after posting I came up with some ideas using some basic if/else for the divide by 0 (though I've yet to try putting it into practice). I'm not sure what you mean by use >0 though?
check that denominator and numerator (or rather divisor, and divider?? forgot the english) are both more than 0, otherwise, how to divide, right? so do a simple if/else...
# 5  
Old 06-01-2009
Quote:
Originally Posted by ghostdog74
check that denominator and numerator (or rather divisor, and divider?? forgot the english) are both more than 0, otherwise, how to divide, right? so do a simple if/else...
Yeah, for once I actually had the basic knowledge to make it work, for some reason the logic just escaped me at the time. I came up with this:

Code:
[root@mys-nycsm-srv01 rtaStats]# cat SOperf 
#!/bin/bash

tail -f /usr/local/mystro/scripts/twctools/rtaStats/strtovrUsage |\
        awk \
        '{if ($4 == "0," || $7 == "0,") \
                print \
                        $1,
                        "Requests = " $4 \
                        "Success = " $7  \
                        "Running Streams = " $10 \
                        "Errors = " $13 \
                        "Percentage... if I tried to get you a percentage the world would explode " \
         ;else \
                print \
                        $1,
                        "Requests = " $4  \
                        "Success = " $7   \
                        "Running Streams = " $10  \
                        "Errors = " $13  \
                        "Percentage = " ($7 / $4 )*100 }'

[root@mys-nycsm-srv01 rtaStats]# SOperf 
20090601-04:03:46 Requests = 5,Success = 5,Running Streams = 17,Errors = 0,Percentage = 100
20090601-04:13:46 Requests = 3,Success = 3,Running Streams = 12,Errors = 0,Percentage = 100
20090601-04:23:47 Requests = 4,Success = 3,Running Streams = 11,Errors = 1,Percentage = 75
20090601-04:33:47 Requests = 1,Success = 0,Running Streams = 7,Errors = 1,Percentage... if I tried to get you a percentage the world would explode 
20090601-04:43:47 Requests = 1,Success = 1,Running Streams = 7,Errors = 0,Percentage = 100
20090601-04:53:47 Requests = 2,Success = 2,Running Streams = 4,Errors = 0,Percentage = 100
20090601-05:03:47 Requests = 4,Success = 4,Running Streams = 4,Errors = 0,Percentage = 100
20090601-05:13:47 Requests = 5,Success = 5,Running Streams = 5,Errors = 0,Percentage = 100
20090601-05:23:47 Requests = 4,Success = 4,Running Streams = 4,Errors = 0,Percentage = 100
20090601-05:33:47 Requests = 4,Success = 4,Running Streams = 5,Errors = 0,Percentage = 100

I'm still confused though. While playing around with the other version of the script I made, I made the minor change:

Code:
#!/bin/bash

printf "Time\t\t\tRequests\tSuccess\tRunning\tError\tPercentage\n"

tail -f  /usr/local/mystro/scripts/twctools/rtaStats/strtovrUsage |\
awk         \
'***** = $1} \
 {req = $4} \
 {suc = $7} \
 {err = $13}\
 {run = $10}\
{print tim "\t" req "\t\t" suc "\t" run "\t" err "\t" (suc / req)*100 }'
[root@mys-nycnm-srv01 ~]# ./xperf 
Time                    Requests        Success Running Error   Percentage
20090601-04:05:51       5,              5,      21,     0,      100
20090601-04:15:51       5,              5,      19,     0,      100
20090601-04:25:51       8,              8,      22,     0,      100
20090601-04:35:51       7,              7,      21,     0,      100
20090601-04:45:51       3,              3,      17,     0,      100
20090601-04:55:51       10,             10,     16,     0,      100
20090601-05:05:51       5,              4,      16,     1,      80
20090601-05:15:52       8,              0,      16,     7,      0
20090601-05:25:52       14,             0,      12,     14,     0
20090601-05:35:52       15,             1,      12,     14,     6.66667

And now it runs perfectly. It even takes care of the divide by zero issue with no additional logic. Admittedly the "END" part was only there because I was cutting and pasting from other things I've pieced together, and not a real understanding of what it means, but could anyone give me a brief explanation of what END was doing?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Alternative to tail -n -0 -F for monitoring live log file

Hello, I have been working on script which need to generate an alert based upon live logs. If string is found then an alert mail must triggered. tail -n -0 -F works fine to redirect the each latest line from live logs file to grep a pattern for matching but it seems to be not working on... (7 Replies)
Discussion started by: ketanraut
7 Replies

2. Shell Programming and Scripting

Piping through grep/awk prevents file write

So, this is weird... I'm running this command: iotop -o -P -k -bt -d 5 I'd like to save the output relelvant to rsyslogd to a file, so I do this: iotop -o -P -k -bt -d 5 | grep rsyslogd >> /var/log/rsyslogd Nothing is written to the file! I can write the full output to the file: ... (2 Replies)
Discussion started by: treesloth
2 Replies

3. Shell Programming and Scripting

How to tail -f logfile. if log file is generate every 1 HR.?

Hello, How to tail -f logfile. if log file is gennerate every 1 HR. I want it works automatically all the time. never changes it by manual. Thank ls -trl CybertonTransaction.* -rw-r--r-- 1 autobot robot 617071 Jul 9 00:02 CybertonTransaction.20130709-00.log -rw-r--r-- 1 autobot ... (12 Replies)
Discussion started by: ooilinlove
12 Replies

4. Shell Programming and Scripting

script to tail file; problem with awk and special characters

Trying to use code that I found to send only new lines out of a log file by doing: while :; do temp=$(tail -1 logfile.out) awk "/$last/{p=1}p" logfile.out #pipe this to log analyzer program last="$temp" sleep 10 done Script works fine when logfile is basic text, but when it contains... (2 Replies)
Discussion started by: moo72moo
2 Replies

5. Shell Programming and Scripting

[Solved] Help piping tail to read STDIN

Hello everybody, I need some help here. I have a log file that gets updated every hour approximately. I want to make some processing on each line which is added in the log file with a program written in PERL. The problem is that I don't see anything when a line is added in the log file. I... (6 Replies)
Discussion started by: Samb95
6 Replies

6. Shell Programming and Scripting

how to tail a log file..

Hi All.. I have a log file in which all the backup information is stored. Now i have written a script which get the last line in the backup log file.. ssh -l ora${sid} ${primaryhost} "tail -1 /oracle/$ORACLE_SID/sapbackup/back$ORACLE_SID.log" However i would like to tail the line last... (4 Replies)
Discussion started by: suri.tyson
4 Replies

7. Shell Programming and Scripting

piping output of tail running in background

Not sure why this does not work in bash: tail -f err.log |& -bash: syntax error near unexpected token `&' I am attempting to continuously read a file that is being updated by doing a "tail -f" on the file and piping the output to stdin which can then be read by the next shell command Thnx (4 Replies)
Discussion started by: anuramdas
4 Replies

8. UNIX for Dummies Questions & Answers

Constantly updating log files (tail -f? grep? awk?)

I have a log file which is continuously added to, called log.file. I'd like to monitor this file, and when certain lines are found, update some totals in another file. I've played around with tail -f, grep, and awk, but can't seem to hit the right note, so to speak. The lines I'm... (0 Replies)
Discussion started by: nortonloaf
0 Replies

9. Shell Programming and Scripting

Bash tail monitor log file

Hi there, I have a problem here that involves bash script since I was noob in that field. Recently, I have to monitor data involve in logs so I just run command tail -f for the monitoring. The logs was generate every hour so I need to quickly change my logs every time the new hour hits according... (2 Replies)
Discussion started by: kriezo
2 Replies

10. Shell Programming and Scripting

tail -f a log file redirected into a new window?

Is this possible? I am attempting to display a new xterm window and tail -f the log file within that new window. I am currently working on a solaris 8 machine if that has any different meaning than the other platforms. As you can see, I am a newbie to this forum and to UNIX. Any help would be... (2 Replies)
Discussion started by: douknownam
2 Replies
Login or Register to Ask a Question