Average across multiple columns group by


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Average across multiple columns group by
# 8  
Old 01-26-2015
Quote:
Originally Posted by Don Cragun
The code you showed us in your 1st post in this thread skips data in the 1st line of your file (which I assumed was intended to skip over a header line). But, I don't see any headers in this sample. Is there a header, or not? If there s a header, should it be copied to the output?

Yes, there is header to be carried over.

Is the number of fields constant in an input file, or can it vary from line to line?

It is constant at 22. The missing values are designated as NA

It looks like there is a leading space in your sample input and output. Is a leading space required in your output?

No leading spaces and none required.

Do you want 2 decimal places in all computed output fields, or do you want values to be printed without decimal places (as in your sample output) in cases where the computed result is an integral value?

Integer values can also be outputted with 2 decimal places, it doesnt matter to add .00 or not.

You say you want to calculate averages for fields 7 through NF, but your sample data also calculates the average for field 6? Is field 6 supposed to be ignored in calculations and removed from the output, or is field 6 to be averaged as well as fields 7 through NF?

Yes, $6 is to be ignored, and doesnt appear in the output.
# 9  
Old 01-26-2015
I want to edit the sample input slightly to accommodate NA missing values, and remove the leading spaces. For a finite number of columns you can of course use arr1, arr2, arr3 etc like you have used arr

Code:
 
a1 b1 c1 d1 e1 12 13 14 15
a1 b1 c1 d1 e1 14 15 16 17
a1 b1 c1 d1 e1 NA 14 15 16
a2 b1 c1 d1 e1 112 113 114 115
a2 b1 c1 d1 e1 114 115 116 117
a2 b1 c1 d1 e1 113 NA 115 116

Output should be


Code:
Code:
a1 b1 c1 d1 e1 13 14 15 16
a2 b1 c1 d1 e1 113 114 115 116


Last edited by senhia83; 01-26-2015 at 08:55 PM.. Reason: formatted sample data
This User Gave Thanks to senhia83 For This Post:
# 10  
Old 01-26-2015
I adjusted your original code to support N counts/accumulators (a pair per each column). I use isnum function from Wikipedia

Code:
 
 awk '
function isnum(x){
        return(x==x+0);
}
{
    if(NR>1) {
        a = $1 $2 $3 $4 $5;
        keys[a] = 1;
        for(I = 7; I <= NF; I++) {
                b = I a;
                arr[b]   += $I;
                if(isnum($I)) {
                        count[b] += 1;
                }
        }
    }
}
END{
        for (key in keys) {
                printf "%-16s ", key;
                for(I = 7; I <= NF; I++) {
                        b = I key;
                        printf "%8.2f ", arr[b] / count[b];
                }
                printf "\n";
        }
}
'

The input file:
Code:
 
 HEADER
a1 b1 c1 d1 e1 f1  1  2  4  5
a1 b1 c1 d1 e1 f1 NA  2  6  7
a1 b1 c1 d1 e1 f1  1  2  5  6
a2 b1 c1 d1 e1 f1 12 13 14 15
a2 b1 c1 d1 e1 f1 14 15 16 17
a2 b1 c1 d1 e1 f1 13 14 15 16

Results
Code:
 
 a2b1c1d1e1          13.00    14.00    15.00    16.00 
a1b1c1d1e1           1.00     2.00     5.00     6.00


Last edited by migurus; 01-26-2015 at 09:10 PM.. Reason: forgot to show input and results
This User Gave Thanks to migurus For This Post:
# 11  
Old 01-26-2015
You could also try something like this:
Code:
awk '
NR == 1 {
	# Note that this copies in the input header to the output, but the
	# output will NOT include field 6 from the input.
	print
	next
}
{	# Gather keys:
	key[k = $1 FS $2 FS $3 FS $4 FS $5]
	# Accumulate counts and data from non-"NA" data fields.
	for(i = 7; i <= NF; i++) {
		if($i != "NA") {
			data[k, i] += $i
			cnt[k, i]++
		}
	}
}
END {	for(k in key) {
		printf("%s ", k)
		for(i = 7; i <= NF; i++)
			if(cnt[k, i])
				printf("%.2f%s", data[k, i] / cnt[k, i],
					(i == NF) ? "\n" : " ")
			else	printf("NA%s", (i == NF) ? "\n" : " ")
	}
}' file

With the following sample input:
Code:
This is a header line
a1 b1 c1 d1 e1 12 13 14 15
a1 b1 c1 d1 e1 14 15 16 17
a1 b1 c1 d1 e1 13 NA 15 16
a2 b1 c1 d1 e1 112 113 114 115
a2 b1 c1 d1 e1 114 115 116 117
a2 b1 c1 d1 e1 113 114 115 NA
a3 b2 c1 d2 e3 110 111 112 NA
a3 b2 c1 d2 e3 110 111 113 NA
a3 b2 c1 d2 e3 110 112 113 NA

the above code produces the output:
Code:
This is a header line
a2 b1 c1 d1 e1 114.00 115.00 116.00
a1 b1 c1 d1 e1 14.00 15.00 16.00
a3 b2 c1 d2 e3 111.33 112.67 NA

Note that with this input, the code migurus suggested will produce output similar to the following:
Code:
a2b1c1d1e1         114.00   115.00   116.00 
a1b1c1d1e1          14.00    15.00    16.00 
a3b2c1d2e3         111.33   112.67 awk: division by zero
 input record number 10, file file
 source line number 23

If someone wants to try the above awk script on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk.
This User Gave Thanks to Don Cragun For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Average of columns

I have files that have the following columns chr pos ref alt sample 1 sample 2 sample 3 chr2 179644035 G A 1,107 0,1 58,67 chr7 151945167 G T 142,101 100,200 500,700 chr13 31789169 CTT CT,C 6,37,8 0,0,0 15,46,89 chr22 ... (3 Replies)
Discussion started by: nans
3 Replies

2. Shell Programming and Scripting

Average of a columns from three files

hello, I have three files in the following order ==> File1 <== 1 20977000 20977000 A C 1.00 0,15 15 45 1 115829313 115829313 G A 0.500 6,7 13 99 ==> File2 <== 1 20977000 20977000 A C 1.00 0,13 13 39 1 115829313 ... (5 Replies)
Discussion started by: nans
5 Replies

3. UNIX for Beginners Questions & Answers

Group by columns and add sum in new columns

Dear Experts, I have input file which is comma separated, has 4 columns like below, BRAND,COUNTRY,MODEL,COUNT NIKE,USA,DUMMY,5 NIKE,USA,ORIGINAL,10 PUMA,FRANCE,DUMMY,20 PUMA,FRANCE,ORIGINAL,15 ADIDAS,ITALY,DUMMY,50 ADIDAS,ITALY,ORIGINAL,50 SPIKE,CHINA,DUMMY,1O And expected output add... (2 Replies)
Discussion started by: ricky1991
2 Replies

4. Shell Programming and Scripting

Average across multiple columns - awk

Hi forum members, I'm trying to get an average of multiple columns in a csv file using awk. A small example of my input data is as follows: cu,u3o8,au,ag -9,20,-9,3.6 0.005,30,-9,-9 0.005,50,10,3.44 0.021,-9,8,3.35 The following code seems to do most of what I want gawk -F","... (6 Replies)
Discussion started by: theflamingmoe
6 Replies

5. Emergency UNIX and Linux Support

Average columns based on header name

Hi Friends, I have files with columns like this. This sample input below is partial. Please check below for main file link. Each file will have only two rows. ... (8 Replies)
Discussion started by: jacobs.smith
8 Replies

6. Shell Programming and Scripting

Match first two columns and average third from multiple files

I have the following format of input from multiple files File 1 24.01 -81.01 1.0 24.02 -81.02 5.0 24.03 -81.03 0.0 File 2 24.01 -81.01 2.0 24.02 -81.02 -5.0 24.03 -81.03 10.0 I need to scan through the files and when the first 2 columns match I... (18 Replies)
Discussion started by: ncwxpanther
18 Replies

7. Shell Programming and Scripting

Get the SUM of TWO columns SEPARATELY by doing GROUP BY on other columns

My File looks like: "|" -> Field separator A|B|C|100|1000 D|E|F|1|2 G|H|I|0|7 D|E|F|1|2 A|B|C|10|10000 G|H|I|0|7 A|B|C|1|100 D|E|F|1|2 I need to do a SUM on Col. 5 and Col.6 by grouping on Col 1,2 & 3 My expected output is: A|B|C|111|11100 (2 Replies)
Discussion started by: machomaddy
2 Replies

8. Shell Programming and Scripting

How to calculate average of two columns and copy into another file?

Hi, I need help with the awk command. I have a folder with aprox 500 files each one with two columns and I want to print in a new file, the average of column 1 and average of column 2 and the name of each file. Input files are: File-1: 100 99 20 99 50 99 50 99 File-2: 200 85... (3 Replies)
Discussion started by: Lokaps
3 Replies

9. Shell Programming and Scripting

Average of columns with values of other column with same name

I have a lot of input files that have the following form: Sample Cq Sample Cq Sample Cq Sample Cq Sample Cq 1WBIN 23.45 1WBIN 23.45 1CVSIN 23.96 1CVSIN 23.14 S1 31.37 1WBIN 23.53 1WBIN 23.53 1CVSIN 23.81 1CVSIN 23.24 S1 31.49 1WBIN 24.55 1WBIN 24.55 1CVSIN 23.86 1CVSIN 23.24 S1 31.74 ... (3 Replies)
Discussion started by: isildur1234
3 Replies

10. UNIX for Dummies Questions & Answers

Taking the average of two columns and printing it on a new column

Hi, I have a space delimited text file that looks like the following: Aa 100 200 Bb 300 100 Cc X 500 Dd 600 X Basically, I want to take the average of columns 2 and 3 and print it in column 4. However if there is an X in either column 2 or 3, I want to print the non-X value. Therefore... (11 Replies)
Discussion started by: evelibertine
11 Replies
Login or Register to Ask a Question