awk - Summing a field based on another field


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers awk - Summing a field based on another field
# 1  
Old 06-19-2009
awk - Summing a field based on another field

So, I need to do some summing. I have an Apache log file with the following as a typical line:

Code:
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET       /apache_pb.gif HTTP/1.0" 200 2326

Now, what I'd like to do is a per-minute sum. So, I can have awk tell me the individual minutes, preserving the dates(since this is a multi-day log):

Code:
awk '{print substr($4,2,17)}' logfile

I can have it sum based on a single date:

Code:
awk 'substr($4,2,17) == "10/Oct/2000:13:55" && $9 ~ /^[0-9]$*/ && $10 ~ /^[0-9]$*/ {sum += $10}' logfile

...but what I'd really like to do is generate a new sum of $10 for each different instance of substr($4,2,17). This can be done by looping-- a foreach/for, etc-- and feeding minutes into the awk one at a time. The problem is that this method reads the file completely every pass through the loop. Ideally, I'd like to do it with a single pass. Any suggestions?
# 2  
Old 06-19-2009
nawk -f tree.awk myLogFile

tree.awk:
Code:
function rindex(str,c)
{
  return match(str,"\\" c "[^" c "]*$")? RSTART : 0
}
{
  idx=substr($4, 2, rindex($4, ":")-1)
  a[idx]+=$NF
}
END {
  for(i in a)
    print i " --> " a[i]
}


Last edited by vgersh99; 06-19-2009 at 09:23 PM.. Reason: a lil' bit more succinct
# 3  
Old 06-19-2009
Quote:
Originally Posted by vgersh99
nawk -f tree.awk myLogFile...
Oh, wow... that's awesome. Thank you very much. A quick followup question of the sort that I usually hate... Smilie If I read that correctly, it doesn't even require that the log entries be in date/time order. Is that so?
# 4  
Old 06-19-2009
Quote:
Originally Posted by treesloth
Oh, wow... that's awesome. Thank you very much. A quick followup question of the sort that I usually hate... Smilie If I read that correctly, it doesn't even require that the log entries be in date/time order. Is that so?
that's right - this's not necessary. On the flip side, however, the output will be in unsorted, random, unspecified order. The latter, can be easily fixed (if necessary - assuming the INCOMING order is temporal).
# 5  
Old 06-19-2009
Quote:
Originally Posted by vgersh99
that's right - there's no necessary. On the flip side, however, the output will be in unsorted, random, unspecified order. The latter, can be easily fixed (if necessary - assuming the INCOMING order is temporal.
Yeah, that's a quick fix. I already have a sorter script for the log file itself... Replacing 4.* with 1.* throughout should do it.

Again, many thanks. This is a fantastic time saver.
# 6  
Old 06-20-2009
Hi vgersh99,

I'm just going through this post and having trouble understanding the code. Could you please explain the rindex function for me? I'm still learning..

Quote:
function rindex(str,c)
{
return match(str,"\\" c "[^" c "]*$")? RSTART : 0
}
Also, why did you use rindex function and not the following. I'm just curious to know and want to improve the way I look at problems.

Code:
idx=substr($4, 2, 17)

Thanks in advance!
# 7  
Old 06-20-2009
Quote:
Originally Posted by King Kalyan
Hi vgersh99,

I'm just going through this post and having trouble understanding the code. Could you please explain the rindex function for me? I'm still learning..
'man nawk' yields:
Code:
     match(s,ere)
           Return the position, in characters, numbering from  1,
           in  string s where the extended regular expression ere
           occurs, or zero if it does not occur  at  all.  RSTART
           will  be  set  to  the starting position (which is the
           same as the returned  value),  zero  if  no  match  is
           found;  RLENGTH  will  be  set  to  the  length of the
           matched string, -1 if no match is found.

Code:
function rindex(str,c)
{
# pass in the string to search (str) and a character to search for (c).
# function will return either a position of 'c' in string 'str' OR '0' if 'c' isn't in
# 'str'. The regex passed to 'match' goes like this:
# "\\" - literal '\' - have to escape it with '\' to preserve the meaning
# followed by char 'c' - resulting in '\c' - need to escape in case 'c' is one
# of the regex chars, e.g. '(', ')', '|' etc
# "[^" - anything BUT the following chars or ranges
# followed by char 'c'
# "]*$" - closing the char list/range - char list repeated 0 or more times (*)
#             anchored to the end of the line ($).
#
# If 'match' found a passed in char, its position will be stored in the
# interval var RSTART - RSTART will be retirned
# If 'match didn't find a passeed in char, 'match' itself will return '0' - 
# therefore a value of '0' will be retuned by the function.
#
# E.g. match("a|b|c|d", "\|[^|]*$")
# Look into the 'match(s,re) description above for more info.
#
return match(str,"\\" c "[^" c "]*$")? RSTART : 0
}

Quote:
Originally Posted by King Kalyan
Also, why did you use rindex function and not the following. I'm just curious to know and want to improve the way I look at problems.

Code:
idx=substr($4, 2, 17)

Thanks in advance!
'cause, I don't like 'magic numbers' and making unnecessary assumptions of the data format IF I can avoid it. E.g., what would happen if we could not guaranteed that the number of minutes/hours was always a TWO digit number: '05' vs '5'. Your '17' would not fly. Whereas looking for the LAST ':' would.
It looked to me like that assuming the leading '[' will always be in the field is safe enough for my taste. On the other hand, if I can avoid counting characters (17) and can identify a 'pattern' and use it to derive what I need, I'd always take this road.
Just my $.02

Last edited by vgersh99; 06-20-2009 at 04:39 PM..
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Problem with getting awk to multiply a field by a value set based on condition of another field

Hi, So awk is driving me crazy on this one. I have searched everywhere and read man, docs and every related post Google can find and still no luck. The actual files I need to run this on are sensitive in nature, but it is the same thing as if I needed to calculate weighted grades for multiple... (15 Replies)
Discussion started by: cotilloe
15 Replies

2. Shell Programming and Scripting

Replacing field based on the value of other field

Hi Team, In the below input file, if I have the value 23,24,25 then for those records 1st field value should get updated from "a" to "b". I also want to pass these values in file as input as it can be done dynamically. Tried awk commands but not getting desired output.Using SunOS 5.10 version.... (14 Replies)
Discussion started by: weknowd
14 Replies

3. Shell Programming and Scripting

awk to update field using matching value in file1 and substring in field in file2

In the awk below I am trying to set/update the value of $14 in file2 in bold, using the matching NM_ in $12 or $9 in file2 with the NM_ in $2 of file1. The lengths of $9 and $12 can be variable but what is consistent is the start pattern will always be NM_ and the end pattern is always ;... (2 Replies)
Discussion started by: cmccabe
2 Replies

4. Shell Programming and Scripting

awk to adjust coordinates in field based on sequential numbers in another field

I am trying to output a tab-delimited result that uses the data from a tab-delimited file to combine and subtract specific lines. If $4 matches in each line then the first matching sequential $6 value is added to $2, unless the value is 1, then the original $2 is used (like in the case of line... (3 Replies)
Discussion started by: cmccabe
3 Replies

5. Shell Programming and Scripting

awk to update value in field based on another field

In the tab-delimeted input file below I am trying to use awk to update the value in $2 if TYPE=ins in bold, by adding the value of HRUN= in italics. In the below since in line 1 TYPE=ins the 117282541 value in $2 has 6 added because that is the value of HRUN=. Hopefully the awk is a start but I... (2 Replies)
Discussion started by: cmccabe
2 Replies

6. Shell Programming and Scripting

How can awk ignore the field delimiter like comma inside a field?

We have a csv file as mentioned below and the requirement is to change the date format in file as mentioned below. Current file (file.csv) ---------------------- empname,date_of_join,dept,date_of_resignation ram,08/09/2015,sales,21/06/2016 "akash,sahu",08/10/2015,IT,21/07/2016 ... (6 Replies)
Discussion started by: gopal.biswal
6 Replies

7. Shell Programming and Scripting

awk to parse field and include the text of 1 pipe in field 4

I am trying to parse the input in awk to include the |gc= in $4 but am not able to. The below is close: awk so far: awk '{sub(/\|]+]++/, ""); print }' input.txt Input chr1 955543 955763 AGRN-6|pr=2|gc=75 0 + chr1 957571 957852 AGRN-7|pr=3|gc=61.2 0 + chr1 970621 ... (7 Replies)
Discussion started by: cmccabe
7 Replies

8. Shell Programming and Scripting

AWK: Pattern match between 2 files, then compare a field in file1 as > or < field in file2

First, thanks for the help in previous posts... couldn't have gotten where I am now without it! So here is what I have, I use AWK to match $1 and $2 as 1 string in file1 to $1 and $2 as 1 string in file2. Now I'm wondering if I can extend this AWK command to incorporate the following: If $1... (4 Replies)
Discussion started by: right_coaster
4 Replies

9. Shell Programming and Scripting

awk, comma as field separator and text inside double quotes as a field.

Hi, all I need to get fields in a line that are separated by commas, some of the fields are enclosed with double quotes, and they are supposed to be treated as a single field even if there are commas inside the quotes. sample input: for this line, 5 fields are supposed to be extracted, they... (8 Replies)
Discussion started by: kevintse
8 Replies

10. Shell Programming and Scripting

Find top N values for field X based on field Y's value

I want to find the top N entries for a certain field based on the values of another field. For example if N=3, we want the 3 best values for each entry: Entry1 ||| 100 Entry1 ||| 95 Entry1 ||| 30 Entry1 ||| 80 Entry1 ||| 50 Entry2 ||| 40 Entry2 ||| 20 Entry2 ||| 10 Entry2 ||| 50... (1 Reply)
Discussion started by: FrancoisCN
1 Replies
Login or Register to Ask a Question