Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Efficient awk way to add numbers in line fields

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Efficient awk way to add numbers in line fields
# 1  
Old 10-14-2017
Efficient awk way to add numbers in line fields
data.now:

Code:
blah1,dah,blaha,sweet,games.log,5297484456,nagios-toin,529748456,on__host=93 SERVICE__ALERT_=51 Warning___The__results__of__service=16 Warning___on__host=92 Auto_save__of__retention__data__completed=1 Warning___Return=68 PASSIVE__SERVICE__CHECK_=53 ,1026--1313,1
blah1,dah,blaha,sweet,games.log,5297484456,nagios-toin,529748456,on__host=93 SERVICE__ALERT_=51 Warning___The__results__of__service=16 Warning___on__host=93 Auto_save__of__retention__data__completed=1 Warning___Return=68 PASSIVE__SERVICE__CHECK_=53 ,1026--1313,1
blah1,dah,blaha,sweet,games.log,5297484456,nagios-toin,529748456,on__host=93 SERVICE__ALERT_=51 Warning___The__results__of__service=16 Warning___on__host=92 Auto_save__of__retention__data__completed=1 Warning___Return=8 PASSIVE__SERVICE__CHECK_=53 ,1026--1313,1
blah1,dah,blaha,sweet,games.log,5297484456,nagios-toin,529748456,on__host=93 SERVICE__ALERT_=51 Warning___The__results__of__service=16 Warning___on__host=99 Auto_save__of__retention__data__completed=1 Warning___Return=68 PASSIVE__SERVICE__CHECK_=53 ,1026--1313,1
blah1,dah,blaha,sweet,games.log,5297484456,nagios-toin,529748456,on__host=93 SERVICE__ALERT_=51 Warning___The__results__of__service=16 Warning___on__host=75 Auto_save__of__retention__data__completed=1 Warning___Return=38 PASSIVE__SERVICE__CHECK_=53 ,1026--1313,1
blah1,dah,blaha,sweet,games.log,5297484456,nagios-toin,529748456,on__host=93 SERVICE__ALERT_=51 Warning___The__results__of__service=16 Warning___on__host=12 Auto_save__of__retention__data__completed=1 Warning___Return=28 PASSIVE__SERVICE__CHECK_=53 ,1026--1313,1


The fields in each line above are separated by "comma". What im interested in is field 9.

Field 9 has a number of values.

What i want to do is two parts:

1. Be able to add up all the values of a specific pattern in all the lines in the datafile. For instance, if i want to know the total value "on__host" on all lines in the data file.

2. Be able to add up all the values from all the patterns on each line, on all the lines in the log.


There was an old data i was working with which only had a single value in the 9th field. that was easy to handle. The data looked like this:

data.prev
Code:
blah1,dah,blaha,sweet,games.log,5297484456,nagios-toin,529748456,53,1026--1313,1

And all I needed to do to add up the values in the 9th field of all the lines was:

Code:
awk -F, 'BEGIN{sum=0} {sum+=$9} END {print sum}' data.prev

The kind of script that would solve this problem for me would look like this:

Code:
#!/bin/sh
Pattern=$1
if [ "${Pattern}" = "allpatterns" ] ; then
   awk should add up all the values in the 9th field of data in data.now
else
   if the user did not specify "allpatterns", then, awk should take the pattern name specified by the user and use that to decide which pattern to add up in the 9th field of all the lines.
    awk -F"," '$9 ~ /'${Pattern}'/ '{do awk magic}'  -- this is just an idea.
fi

Last edited by SkySmart; 10-14-2017 at 10:27 PM..
# 2  
Old 10-14-2017
Try using the = as the field sep.

adding whole line
Code:
awk -F '='  { for (sum=0, i=2; i++; i<=NF) {
                         sum += int($(i)  )
                         if(i==NF) {print sum}       } 
           } ' somefile

You should try to modify the code above to print line field numbers and then use the END{} function to show the sum of all the lines.
# 3  
Old 10-15-2017
Hi SkySmart,
As you know well, it always helps those of us who want to help you if we know (1) what operating system and shell you're using, and (2) if you actually show us the output you are hoping to produce from the sample input you provided.

Without (1) you are likely to have people who want to help you waste their time on suggestions that won't work in your environment. Please tell us what operating system and shell you're using.

Without (2), especially with the sparse description of what you are trying to do, you leave people guessing at what output you're trying to produce. Please show us the output you are hoping to produce from your sample input file.
# 4  
Old 10-15-2017
Quote:
Originally Posted by Don Cragun
Hi SkySmart,
As you know well, it always helps those of us who want to help you if we know (1) what operating system and shell you're using, and (2) if you actually show us the output you are hoping to produce from the sample input you provided.

Without (1) you are likely to have people who want to help you waste their time on suggestions that won't work in your environment. Please tell us what operating system and shell you're using.

Without (2), especially with the sparse description of what you are trying to do, you leave people guessing at what output you're trying to produce. Please show us the output you are hoping to produce from your sample input file.
sorry. i thought i provided more than enough information in my original post. but i have no issues providing more.

this script is expected to run on all unix systems. The shell i'm using is /bin/sh.

For the first part of my request, if all the values of all the patterns on all the lines in the data file are added up, the output should be just the resulting sum of the numbers..i.e 504 (just an arbitrary number i picked).

For the second part of my request, if the values of a specific pattern are added up, the output should be just the resulting sum of the values for that specific pattern...i.e. "on__host=400".
# 5  
Old 10-15-2017
Hi try something like:
Code:
awk -F, -v pat="$Pattern" '
  {
    n=split($9,F," ")
    for(i=1; i<=n; i++) {
      if(F[i]~"^" pat) {
        split(F[i],VAL,/=/)
        t+=VAL[2]
      }
    }
  }

  END {
    mess=pat "=" t
    sub(/^=/,x,mess)
    print mess
  }
' somefile

If $Pattern is empty it just renders a total of all fields in $9
This User Gave Thanks to Scrutinizer For This Post:
SkySmart
# 6  
Old 10-16-2017
Quote:
Originally Posted by Scrutinizer
Hi try something like:
Code:
awk -F, -v pat="$Pattern" '
  {
    n=split($9,F," ")
    for(i=1; i<=n; i++) {
      if(F[i]~"^" pat) {
        split(F[i],VAL,/=/)
        t+=VAL[2]
      }
    }
  }

  END {
    mess=pat "=" t
    sub(/^=/,x,mess)
    print mess
  }
' somefile

If $Pattern is empty it just renders a total of all fields in $9

this worked beautifully. i modified it so that it allows me to choose which lines i want it to read, but instead of showing me the total of the numbers, its printing out all the lines.

What is wrong with the modified code below:

Code:
awk -F, -v pat="$Pattern" 'BEGIN{t=0} /'Sun' 'Oct' '15' '22':[0-9][0-9]:[0-9][0-9] '2017',/
  {
    n=split($9,F," ")
    for(i=1; i<=n; i++) {
      if(F[i]~"^" pat) {
        split(F[i],VAL,/=/)
        t+=VAL[2]
      }
    }
  }

  END {
    mess=pat "=" t
    sub(/^=/,x,mess)
    print mess
  }
' somefile

the above code scans the specific lines i want, but instead of calculating the numbers in field 9, it prints them (the lines) out, and prints an incorrect total number at the end.

i.e. results:

Code:
0,data,Sun Oct 15 22:01:23 2017,6906,/var/log/catalina.out,524K,data,529086,Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=0 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0 The_file_or_directory_is_corrupted_and_unreadable=0 ,6894 6906,148       
0,data,Sun Oct 15 22:06:23 2017,6906,/var/log/catalina.out,524K,data,529086,emxCommonAppInitialization__Error_while_initializing=0  INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0  java_lang_NoClassDefFoundError=0  java_lang_OutOfMemoryError=0  Master_Item_Service_is_down=0  The_file_or_directory_is_corrupted_and_unreadable=0,6906 6906,448  
0,data,Sun Oct 15 22:11:23 2017,6914,/var/log/catalina.out,524K,data,529580,Master_Item_Service_is_down=0 java_lang_NoClassDefFoundError=0 java_lang_OutOfMemoryError=0 emxCommonAppInitialization__Error_while_initializing=0 INFO__Stopping_Coyote_HTTP_1_1_on_http_8080=0 The_file_or_directory_is_corrupted_and_unreadable=0 ,6906 6914,100
5

notice the 5 at the end. not sure where its getting 5 from.
Last edited by SkySmart; 10-16-2017 at 03:00 AM..
# 7  
Old 10-16-2017
Your code prints the lines for 10pm on Sunday, October 15, 2017 (from the 2nd statement in the first line of your awk code. I assume that the value you seem to want from those lines total up to zero. But the calculations in your script (from the statement on lines 2 through 10 in your awk script) are performed on every line in the file; not just those that you print (since the condition on that statement is the empty expression).

Again, you say the output is not what you want, but do not clearly show us what output you are trying to produce.

And, you have not shown us the input that you are feeding this script. So, we have no idea where the 5 is coming from either. If your input file contained the line:
Code:
,,,,,,=5

plus the first three lines shown in your output and $pattern expands to an empty string, you would likely get the output you showed us.
This User Gave Thanks to Don Cragun For This Post:
SkySmart
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to identify empty fields in line

I am trying to use awk to identify and print out records in fields that are empty along with which line they are in. I hope the awk below is close, it runs but nothing results. Thank you :). awk awk -F'\t' 'FNR==NR ~ /^*$/ { print "NR is empty" }' file file 123 GOOD ID 45... (3 Replies)
Discussion started by: cmccabe
3 Replies

2. Shell Programming and Scripting

awk - set numbers [ 1 ... n] from the 6 line

Hi, i have a file, where measurement-data is stored in the first column. The file has also a header of 5 lines. I want to set counting up numbers in front of any particular measurement-value; should start at the 6. line with starting number 1. i try to solve it with ... awk 'NR > 6 { print... (6 Replies)
Discussion started by: IMPe
6 Replies

3. Shell Programming and Scripting

How to add line numbers (multiples of 5: 0,5,10,15,20) to a text file?

Hi, I need to number the lines in my text file. I know how to do this with standard numbering (1,2,3,4, etc) but I need to count in multiples of 5, beginning 0,5,10,15... example existing file: abcd efg hijklm nopqrs desired output 0 abcd 5 efg 10 hijklm 15 ... (11 Replies)
Discussion started by: livbaddeley
11 Replies

4. Shell Programming and Scripting

awk (or other) script that assigns fields from a line to multiple variables

Hey all, Unfortunately I have only basic knowledge of awk and/or scripting. If I have a file with lines that can look similar to this: Name=line1 Arg1=valueA Arg2=valueB Arg3=valueC Name=line2 Arg1=valueD Name=line3 Arg1=valueE Arg3=valueF Name=line4 Arg2=valueG ... (4 Replies)
Discussion started by: Rike255
4 Replies

5. Shell Programming and Scripting

Using Awk to Add Numbers

echo "0.1 2.0 0.4 2.0 4.3 1.0 6.0 9.0" | awk 'BEGIN {total=0} {total += $1} END {print total}' I want to add the above output from the echo command, but i can't figure this out. The output above always spits out inaccurate numbers. can someone please provide me with a one liner similar to... (4 Replies)
Discussion started by: SkySmart
4 Replies

6. Shell Programming and Scripting

AWK multiple line fields sorting

I have a bash script which takes a log file with each record separated by a #. The records have multiple fields but field $1 is always the date and time. When the script is run it prints the record just fine from oldest to newest. I need to have records print out from newest first. Here is the... (7 Replies)
Discussion started by: numele
7 Replies

7. Shell Programming and Scripting

Add to constant fields at the end of every line

Hi, I want to add two fields with values '1000' and 'XYZ-1234' at the end of every line in a comma delimited file. Should I use any command in a loop to add the fields or using any single command Shall I acheive it? Kindly help me in code. Thanks, Poova. (6 Replies)
Discussion started by: poova
6 Replies

8. Shell Programming and Scripting

how to add line numbers in text file

Hi all How to add line numbers in text file.. ex abcd cdef result 1. abcd 2. cdef thx in advance (4 Replies)
Discussion started by: suryanarayana
4 Replies

9. UNIX for Advanced & Expert Users

Add line numbers to end of each line

Hi i would like to add line numbers to end of each line in a file. I am able to do it in the front of each line using sed, but not able to add at the end of the file. Can anyone suggest The following code adds line number to start of each line sed = filename | sed 'N;s/\n/\t/' how can i... (5 Replies)
Discussion started by: rudoraj
5 Replies

10. Shell Programming and Scripting

add line numbers

Hello.. I have got one file ... I want to add line numbers with space form starting to ending.. for example...if the file is -------------------------- sand sorcd 2345 345 recds 234 234 5687 yeres 568 988 erfg4 67 -------------------------- I need the output ... (4 Replies)
Discussion started by: esham
4 Replies
Login or Register to Ask a Question