The UNIX and Linux Forums  

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
Is it better to grep and pipe to awk, or to seach with awk itself Is it better to grep and pipe to awk, or to seach with awk itself
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Seach for Record xtravel Shell Programming and Scripting 5 07-03-2007 11:48 AM
Grep for NULL in a pipe delimited file sureshg_sampat Shell Programming and Scripting 5 11-21-2006 06:15 AM
Pipe Data From Grep Into A File katinicsdad Shell Programming and Scripting 4 09-08-2006 12:20 PM
Filter results through pipe with grep ckandreou UNIX for Dummies Questions & Answers 1 07-10-2006 03:04 PM
pipe help bb666 High Level Programming 5 02-26-2002 04:07 PM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Bulgarian Greek Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 10-07-2008
DeCoTwc DeCoTwc is offline
Registered User
  
 

Join Date: Mar 2008
Location: NYC
Posts: 77
Is it better to grep and pipe to awk, or to seach with awk itself
This may just be a lack of experience talking, but I always assumed that when possible it was better to use a commands built in abilities rather than to pipe to a bunch of commands. I wrote a (very simple) script a while back that was meant to pull out a certain error code, and report back what piece of equipment had thrown it:


Code:
#!/usr/bin/bash
######################################################
# Program:      problemboxes.sh 
# Date Created: 14 May 2008
# Developer:    Darrell S. **** (Digital Sys. Admin)
# Description:  Generates lists of boxes that have thrown excess error 18's 
# Last Updated: 7 Aug 2008
######################################################
clear
user=`echo $UID`
if 
[ $user != 0 ]; then
echo "you must be root to use this script";exit
else

clear
echo "generating list of boxes that have thrown more than 5 error 18's today. This may take a moment, please wait"

grep "AddResCnf response (0x12) is not 'OK'" /usr/local/n2bb/log/n2bb.log|awk '{print $10}'|tr -d "'"|cut -c 1-12|sort|uniq -c|awk '{ if ($1 > 5) print $1,$2}'
fi

Last night while bored I decided to try simplifying the script and came up with:


Code:
#!/usr/bin/bash
######################################################
# Program:      problemboxes.sh 
# Date Created: 14 May 2008
# Developer:    Darrell S. **** (Digital Sys. Admin)
# Description:  Generates lists of boxes that have thrown excess error 18's
# Last Updated: 5 October 2008
######################################################
clear
user=`echo $UID`
if 
[ $user != 0 ]; then
echo "you must be root to use this script";exit
else
clear
echo "generating list of boxes that have thrown more than 15 error 18's today. This may take a moment, please wait"
awk -F\' '/0x12/ {print $2}' /usr/local/n2bb/log/n2bb.log|cut -c 1-12|sort|uniq -c|awk '{ if ($1 > 15) print $1,$2}'
fi

What seems odd to me is, that while both return the same results, the one that searches with awk, takes considerably longer (granted only ~3 seconds right now, but that's because the log rolled at midnight) than the one that uses grep.

I have several scripts that use basically the same logic, just sort the information later, and as they tend to use up a lot of processor power (the logs these crawl are pretty big) I'd like to make them as efficient as possible.

In case it matters this is what the overall log file tends to look like:


Code:
root@bms02-twc-NM-newyork-ny:/usr/local/n2bb/log# tail n2bb.log
2008/10/07 04:27:04.348 GMT(10/07 00:27:04 -0400) INFO       SESSIONGW  N2BBSessionGateway_impl.DsmccMsgListener(): Received 'AddResCnf' message for session '001bd744aca600000f12'
2008/10/07 04:27:04.348 GMT(10/07 00:27:04 -0400) INFO       SESSIONGW  N2BBSessionGateway_impl.MsgHndlrThread(): Processing 'SvrAddResCnf' message for session '001bd744aca600000f12'
2008/10/07 04:27:04.455 GMT(10/07 00:27:04 -0400) INFO       SESSIONGW  N2BBSessionGateway_impl.msg_handler_SvrAddResCnf(): Sending SvrSetupRsp for session '001bd744aca600000f12'
2008/10/07 04:27:04.455 GMT(10/07 00:27:04 -0400) INFO       SESSIONGW  N2BBSessionGateway_impl.msg_handler_SvrAddResCnf(): Successfully set up session '001bd744aca600000f12'
2008/10/07 04:27:05.516 GMT(10/07 00:27:05 -0400) INFO       SESSIONGW  N2BBSessionGateway_impl.DsmccMsgListener(): Received 'SvrRelInd' message for session '0001a6fc0db015962b30'
2008/10/07 04:27:05.516 GMT(10/07 00:27:05 -0400) INFO       SESSIONGW  N2BBSessionGateway_impl.MsgHndlrThread(): Processing 'SvrRelInd' message for session '0001a6fc0db015962b30'
2008/10/07 04:27:05.516 GMT(10/07 00:27:05 -0400) INFO       SESSIONGW  N2BBSessionGateway_impl.releaseSession(): Sending SvrRlsRsp for session '0001a6fc0db015962b30'
2008/10/07 04:27:05.766 GMT(10/07 00:27:05 -0400) INFO       SESSIONGW  N2BBSessionGateway_impl.DsmccMsgListener(): Received 'SvrRelInd' message for session '00e0366d16b605c58bea'
2008/10/07 04:27:05.766 GMT(10/07 00:27:05 -0400) INFO       SESSIONGW  N2BBSessionGateway_impl.MsgHndlrThread(): Processing 'SvrRelInd' message for session '00e0366d16b605c58bea'
2008/10/07 04:27:05.767 GMT(10/07 00:27:05 -0400) INFO       SESSIONGW  N2BBSessionGateway_impl.releaseSession(): Sending SvrRlsRsp for session '00e0366d16b605c58bea'
root@bms02-twc-NM-newyork-ny:/usr/local/n2bb/log#

And this is the error that I'm looking to pull information from:


Code:
root@bms02-twc-NM-newyork-ny:/usr/local/n2bb/log# grep 0x12 n2bb.log
2008/10/07 04:01:45.592 GMT(10/07 00:01:45 -0400) ERROR      SESSIONGW  N2BBSessionGateway_impl.msg_handler_SvrAddResCnf(): Session '00e03665f4412ab15563' failed:  Unable to get resources in service group 11627.  AddResCnf response (0x12) is not 'OK'
2008/10/07 04:01:48.994 GMT(10/07 00:01:48 -0400) ERROR      SESSIONGW  N2BBSessionGateway_impl.msg_handler_SvrAddResCnf(): Session '00e03665f4412ab15564' failed:  Unable to get resources in service group 11627.  AddResCnf response (0x12) is not 'OK'
  #2 (permalink)  
Old 10-07-2008
cfajohnson's Avatar
cfajohnson cfajohnson is offline Forum Advisor  
Shell programmer, author
  
 

Join Date: Mar 2007
Location: Toronto, Canada
Posts: 2,362
Quote:
Originally Posted by DeCoTwc View Post
This may just be a lack of experience talking, but I always assumed that when possible it was better to use a commands built in abilities rather than to pipe to a bunch of commands. I wrote a (very simple) script a while back that was meant to pull out a certain error code, and report back what piece of equipment had thrown it:


Code:
#!/usr/bin/bash
######################################################
# Program:      problemboxes.sh 
# Date Created: 14 May 2008
# Developer:    Darrell S. **** (Digital Sys. Admin)
# Description:  Generates lists of boxes that have thrown excess error 18's 
# Last Updated: 7 Aug 2008
######################################################
clear
user=`echo $UID`

Why are you using command substitution instead of a straight assignment? In all shells except ksh93, it forks a new process and is almost as slow as an external command. Use:


Code:
user=$UID

Quote:

[snip]

What seems odd to me is, that while both return the same results, the one that searches with awk, takes considerably longer (granted only ~3 seconds right now, but that's because the log rolled at midnight) than the one that uses grep.

The search code in grep is much faster than that in all versions of awk except mawk, but it will not make much difference except on very large files.

The authors of AWK, in their book, The AWK Programming Language, recommend using grep to search and piping the results though awk for processing rather than doing it all in AWK.
  #3 (permalink)  
Old 10-07-2008
DeCoTwc DeCoTwc is offline
Registered User
  
 

Join Date: Mar 2008
Location: NYC
Posts: 77
Quote:
Originally Posted by cfajohnson View Post

Why are you using command substitution instead of a straight assignment? In all shells except ksh93, it forks a new process and is almost as slow as an external command. Use:


Code:
user=$UID


The search code in grep is much faster than that in all versions of awk except mawk, but it will not make much difference except on very large files.

The authors of AWK, in their book, The AWK Programming Language, recommend using grep to search and piping the results though awk for processing rather than doing it all in AWK.
The UID thing...because I'm for lack of a better word...a newb.

Thanks for the info regarding the awk V. grep. The log I'm searching gets to be rather large as it only rolls every 12 hours. And even 45 minutes into it's cycle there was already a 3 second difference in running with grep as opposed to grep.

My second question is an extension of the first. Is there any benefit to giving grep more or less to search for?

is it better to grep for 0x12, as that's the error code, or to grep for "AddResCnf response (0x12) is not 'OK'" which is the entire error? In my mind I could think of logical reasons why I could think of why a longer search term is better than a short one...but as I said, I'm kind of a new.
  #4 (permalink)  
Old 10-07-2008
cfajohnson's Avatar
cfajohnson cfajohnson is offline Forum Advisor  
Shell programmer, author
  
 

Join Date: Mar 2007
Location: Toronto, Canada
Posts: 2,362

I don't think there'd be much difference, but try it and see.
  #5 (permalink)  
Old 10-07-2008
DeCoTwc DeCoTwc is offline
Registered User
  
 

Join Date: Mar 2008
Location: NYC
Posts: 77
misread prior post
Last edited by DeCoTwc; 10-07-2008 at 03:53 PM.. Reason: misread
Closed Thread

Bookmarks

« RE: exit value | unsetting OPTINT in getopts »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Hybrid Mode Hybrid Mode
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 06:06 AM.

The UNIX and Linux Forums - Learn UNIX and UNIX Commands RSS Feeds - Contact Us - The UNIX and Linux Forums - Learn UNIX and UNIX Commands - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0