Is it better to grep and pipe to awk, or to seach with awk itself
This may just be a lack of experience talking, but I always assumed that when possible it was better to use a commands built in abilities rather than to pipe to a bunch of commands. I wrote a (very simple) script a while back that was meant to pull out a certain error code, and report back what piece of equipment had thrown it:
Last night while bored I decided to try simplifying the script and came up with:
What seems odd to me is, that while both return the same results, the one that searches with awk, takes considerably longer (granted only ~3 seconds right now, but that's because the log rolled at midnight) than the one that uses grep.
I have several scripts that use basically the same logic, just sort the information later, and as they tend to use up a lot of processor power (the logs these crawl are pretty big) I'd like to make them as efficient as possible.
In case it matters this is what the overall log file tends to look like:
And this is the error that I'm looking to pull information from:
This may just be a lack of experience talking, but I always assumed that when possible it was better to use a commands built in abilities rather than to pipe to a bunch of commands. I wrote a (very simple) script a while back that was meant to pull out a certain error code, and report back what piece of equipment had thrown it:
Why are you using command substitution instead of a straight assignment? In all shells except ksh93, it forks a new process and is almost as slow as an external command. Use:
Quote:
[snip]
What seems odd to me is, that while both return the same results, the one that searches with awk, takes considerably longer (granted only ~3 seconds right now, but that's because the log rolled at midnight) than the one that uses grep.
The search code in grep is much faster than that in all versions of awk except mawk, but it will not make much difference except on very large files.
The authors of AWK, in their book, The AWK Programming Language, recommend using grep to search and piping the results though awk for processing rather than doing it all in AWK.
Why are you using command substitution instead of a straight assignment? In all shells except ksh93, it forks a new process and is almost as slow as an external command. Use:
The search code in grep is much faster than that in all versions of awk except mawk, but it will not make much difference except on very large files.
The authors of AWK, in their book, The AWK Programming Language, recommend using grep to search and piping the results though awk for processing rather than doing it all in AWK.
The UID thing...because I'm for lack of a better word...a newb.
Thanks for the info regarding the awk V. grep. The log I'm searching gets to be rather large as it only rolls every 12 hours. And even 45 minutes into it's cycle there was already a 3 second difference in running with grep as opposed to grep.
My second question is an extension of the first. Is there any benefit to giving grep more or less to search for?
is it better to grep for 0x12, as that's the error code, or to grep for "AddResCnf response (0x12) is not 'OK'" which is the entire error? In my mind I could think of logical reasons why I could think of why a longer search term is better than a short one...but as I said, I'm kind of a new.
In the below awk to add a sort by smallest to largest should it be added after the END? Thank you :).
BEGIN {
FS="*"
}
# Read search terms from file1 into 's'
FNR==NR {
s
next
}
{
# Check if $5 matches one of the search terms
for(i in s) {
if($5 ~ i) {
... (4 Replies)
Hi
i need a help for this.
the output of a command is like below
1 /tmp/x
2.2K /tmp/y
3.2k /tmp/z
1G /tmp/a/b
2.2G /tmp/c
3.4k /tmp/d
Now i need to grep for the paths which are in GB..like below
1G /tmp/a/b
2.2G /tmp/c
pls suggest me, how can i... (12 Replies)
Hey fellas,
I wrote an script which its output is like this:
a 1 T
a 1 T
a 2 A
b 5 G
b 5 G
b 5 G
I wanna print $1 $2 and the total number of $2 value as the third column and after that $3. Sth like this:
a 1 2 T
a 2 1 A
b 5 3 G
I know how to do it with a given input... (4 Replies)
Hi,
I'm testing nginx-cache-purge and notice that grep searches all file content, and since the cache key is in the second line I would like to limit grep searching.
This is the script I'm using:
github.com/perusio/nginx-cache-purge
The line I would like to change and limit grep search... (5 Replies)
<tr><th align=right valign=top>Faulty_Part</th><td align=left valign=top>readhat version 6.0</td></tr> <tr><th align=right valign=top>Submit_Date</th><td align=left valign=top>2011-04-28 02:08:02</td></tr> .......(a long string)
I want to get all the field between "left valign=top>" and "... (2 Replies)
Hi!
If I'm trying something like:
echo "hello world" | myvar=`awk -F "world" '{print $1}'`
echo $myvar
myvar is always empty :confused:
I googled for houres now and don't understand why it isn't working...
Trying it in normal bash.
Can someone explain it to me so I can say "Of course!... (8 Replies)
Hello,
I trying to extract text that is surrounded by xml-tags. I tried this
cat tst.xml | egrep "<SERVER>.*</SERVER>" |sed -e "s/<SERVER>\(.*\)<\/SERVER>/\1/"|tr "|" " "
which works perfect, if the start-tag and the end-tag are in the same line, e.g.:
<tag1>Hello Linux-Users</tag1>
... (5 Replies)
This script is supposed to find out if tomcat is running or not.
#!/bin/sh
if netstat -a | grep `grep ${1}: /tomcat/bases | awk -F: '{print $3}'` > /dev/null
then
echo Tomcat for $1 running
else
echo Tomcat for $1 NOT running
fi
the /tomcat/bases is a file that... (2 Replies)
Can anyone help me how I will extract all lines in a file where the word "worker" or "co-worker" in 2nd column exist. There are also word in 2nd column like "workers" or "worker2" but I don't want to display those lines.
Appreciate any help in advance! Thank you! (5 Replies)