awk - Summing a field based on another field

treesloth · #1 (**permalink**) 06-19-2009

So, I need to do some summing. I have an Apache log file with the following as a typical line:

Code:

127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET       /apache_pb.gif HTTP/1.0" 200 2326

Now, what I'd like to do is a per-minute sum. So, I can have awk tell me the individual minutes, preserving the dates(since this is a multi-day log):

Code:

awk '{print substr($4,2,17)}' logfile

I can have it sum based on a single date:

Code:

awk 'substr($4,2,17) == "10/Oct/2000:13:55" && $9 ~ /^[0-9]$*/ && $10 ~ /^[0-9]$*/ {sum += $10}' logfile

...but what I'd really like to do is generate a new sum of $10 for each different instance of substr($4,2,17). This can be done by looping-- a foreach/for, etc-- and feeding minutes into the awk one at a time. The problem is that this method reads the file completely every pass through the loop. Ideally, I'd like to do it with a single pass. Any suggestions?

vgersh99 · #2 (**permalink**) 06-19-2009

nawk -f tree.awk myLogFile

tree.awk:

Code:

function rindex(str,c)
{
  return match(str,"\\" c "[^" c "]*$")? RSTART : 0
}
{
  idx=substr($4, 2, rindex($4, ":")-1)
  a[idx]+=$NF
}
END {
  for(i in a)
    print i " --> " a[i]
}

treesloth · #3 (**permalink**) 06-19-2009

Quote:

Originally Posted by vgersh99

nawk -f tree.awk myLogFile...

Oh, wow... that's awesome. Thank you very much. A quick followup question of the sort that I usually hate...

If I read that correctly, it doesn't even require that the log entries be in date/time order. Is that so?

vgersh99 · #4 (**permalink**) 06-19-2009

Quote:

Originally Posted by treesloth

Oh, wow... that's awesome. Thank you very much. A quick followup question of the sort that I usually hate...

If I read that correctly, it doesn't even require that the log entries be in date/time order. Is that so?

that's right - this's not necessary. On the flip side, however, the output will be in unsorted, random, unspecified order. The latter, can be easily fixed (if necessary - assuming the INCOMING order is temporal).

treesloth · #5 (**permalink**) 06-19-2009

Quote:

Originally Posted by vgersh99

that's right - there's no necessary. On the flip side, however, the output will be in unsorted, random, unspecified order. The latter, can be easily fixed (if necessary - assuming the INCOMING order is temporal.

Yeah, that's a quick fix. I already have a sorter script for the log file itself... Replacing 4.* with 1.* throughout should do it.

Again, many thanks. This is a fantastic time saver.

King Kalyan · #6 (**permalink**) 06-20-2009

Hi vgersh99,

I'm just going through this post and having trouble understanding the code. Could you please explain the rindex function for me? I'm still learning..

Quote:

function rindex(str,c)
{
return match(str,"\\" c "[^" c "]*$")? RSTART : 0
}

Also, why did you use rindex function and not the following. I'm just curious to know and want to improve the way I look at problems.

Code:

idx=substr($4, 2, 17)

Thanks in advance!

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
split large file based on field criteria	asriva	Shell Programming and Scripting	6	06-22-2009 10:41 AM
Sort alpha on 1st field, numerical on 2nd field (sci notation)	FrancoisCN	Shell Programming and Scripting	1	06-12-2009 10:45 AM
Find top N values for field X based on field Y's value	FrancoisCN	Shell Programming and Scripting	1	05-29-2009 09:57 AM
Split file based on field	s_adu	Shell Programming and Scripting	6	05-07-2009 02:08 PM
add lines automatically based on a field on another file	melanie_pfefer	Shell Programming and Scripting	0	07-24-2008 02:59 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by