Count number of unique values in each column of array
What is an efficient way of counting the number of unique values in a 400 column by 1000 row array and outputting the counts per column, assuming the unique values in the array are:
A, B, C, D
In other words the output should look like:
Thanks in advance
Last edited by Scrutinizer; 01-14-2018 at 02:03 PM..
Reason: added example; mod: code tags
Given the code suggested by Chubler_XL and Yoda, it is hard to imagine that anything is different in the way counts are accumulated for column numbers that are multiples of 13.
Can you provide us with sample data that demonstrates the inaccurate counts that you have reported?
What operating system (including release number) are you using?
Which version of awk are you using?
This User Gave Thanks to Don Cragun For This Post:
Given the code suggested by Chubler_XL and Yoda, it is hard to imagine that anything is different in the way counts are accumulated for column numbers that are multiples of 13.
Can you provide us with sample data that demonstrates the inaccurate counts that you have reported?
What operating system (including release number) are you using?
Which version of awk are you using?
Sure, for Awk I have GNU Awk 4.1.3, and for OS I have Ubuntu 16.04.3 LTS, and I have attached the actual text file I used. Also, I stand corrected it seems that it is not multiples of 13 but many more columns at random that are off.
When I slightly modify the code suggested by Chubler_XL to be:
Code:
awk '
{
mc = NF > mc ? NF : mc
for(i=NF; i; i--) {
T[$i]
C[i FS $i]++
}
}
END {
printf "Value"
for(i=1; i<=mc;i++) printf "\tCOL%d",i
for(v in T) {
printf "\n%s", v
for(i=1; i<=mc;i++) printf "\t%d",C[i FS v]
}
printf "\n"
}' a2.txt
and store this in a file named Chubler_XL, make it executable and run the command:
Code:
./Chubler_XL > Chubler_XL.out
and I slightly modify the code suggested by Yoda to be:
Code:
awk '
BEGIN {
n = split ( "./. 0/0 0/1 1/1", T )
}
{
for ( i = 1; i <= NF; i++ )
R[i FS $i] += 1
}
END {
printf "VAL\t"
for ( i = 1; i <= NF; i++ )
printf "COL%d\t", i
printf "\n"
for ( j = 1; j <= n; j++ )
{
printf "%s\t", T[j]
for ( i = 1; i <= NF; i++ )
printf "%d\t", R[i FS T[j]]
printf "\n"
}
}
' a2.txt
and store this in a file named Yoda, make it executable and run the command:
Code:
./Yoda > Yoda.out
and I write the code:
Code:
awk -v line_count="$(wc -l < a2.txt)" '
function check() {
printf("Checking fields 2 through %d in file: %s\n", NF, f)
for(i = 2; i <= NF; i++)
if(c[i] != line_count)
printf("file %s: field %d count %d\n", f, i, c[i])
split("", c)
}
FNR == 1 {
line_count += 0
if(f == "")
printf("Evaluating output produced from %d lines in a2.txt\n",
line_count)
else
check()
f = FILENAME
next
}
{ for(i = 2; i <= NF; i++)
c[i] += $i
}
END { check()
}' *.out
and store that in a file named counter, make it executable, and run it, I get the output:
Code:
Evaluating output produced from 83 lines in a2.txt
Checking fields 2 through 305 in file: Chubler_XL.out
Checking fields 2 through 305 in file: Yoda.out
which shows that the sums of the values for each of the 304 fields does indeed equal the number of lines found in the file you attached in post #6.
I see no indication that either of these suggestions is producing results that are incorrect although neither of them produce output that is at all close to the output you showed us in post #4. I do note that the output you showed us in post #4 only shows output for the three values "0/0", "0/1", and "1/1"; but the data in a2.txt also includes some entries with the value "./." which is included in the output produced by the code Chubler_XL suggested and in the output produced by the code Yoda suggested (after changing it to look for those four values instead of the values, "A", "B", "C", and "D" that you said were included as values in your statements in post #1.
If you'd like to show us the code you used to produce the output for the 1st 26 columns you showed us in post #4, maybe we can help you explain why that code failed to correctly interpret the output produced by Chubler_XL's code or Yoda's code.
This User Gave Thanks to Don Cragun For This Post:
Hello,
I am trying to count unique rows in my file based on 4 columns (2-5) and to output its frequency in a sixth column. My file is tab delimited
My input file looks like this:
Colum1 Colum2 Colum3 Colum4 Coulmn5
1.1 100 100 a b
1.1 100 100 a c
1.2 200 205 a d
1.3 300 301 a y
1.3 300... (6 Replies)
Hi,
tab-separated input:
blabla_1 A,B,C,C
blabla_2 A,E,G
blabla_3 R,Q,A,B,C,R,Q
output:
blabla_1 3
blabla_2 3
blabla_3 5
After splitting $2 in an array, I am trying to store the number of unique elements in a variable, but have some difficulties resetting the variable to 0 before... (6 Replies)
Hello experts,
I am converting a number into its binary output as :
read n
echo "obase=2;$n" | bc
I wish to count the maximum continuous occurrences of the digit 1.
Example :
1. The binary equivalent of 5 = 101. Hence the output must be 1.
2. The binary... (3 Replies)
Hello Team,
I need your help on the following:
My input file a.txt is as below:
3330690|373846|108471
3330690|373846|108471
0640829|459725|100001
0640829|459725|100001
3330690|373847|108471
Here row 1 and row 2 of column 1 are identical but corresponding column 2 value are... (4 Replies)
Hi, I have tab-deliminated data similar to the following:
dot is-big 2
dot is-round 3
dot is-gray 4
cat is-big 3
hot in-summer 5
I want to count the frequency of each individual "unique" value in the 1st column. Thus, the desired output would be as follows:
dot 3
cat 1
hot 1
is... (5 Replies)
Hi !
input:
A|B|C|D
A|F|C|E
A|B|I|C
A|T|I|B
As the title of the thread says, I would need to get:
1|3|2|4
I tried different variants of this command, but I don't manage to obtain what I need:
gawk 'BEGIN{FS=OFS="|"}{for(i=1; i<=NF; i++) a++} END {for (b in a) print b}' input
... (2 Replies)
Hi
I have the following info in a file -
<Cell id="25D"/>
<Cell id="26A"/>
<Cell id="26B"/>
<Cell id="26C"/>
<Cell id="27A"/>
<Cell id="27B"/>
<Cell id="27C"/>
<Cell id="28A"/>
I would like to know how would you go about counting all... (4 Replies)
Could anybody help with this?
I have input below .....
david,39
david,39
emelie,40
clarissa,22
bob,42
bob,42
tim,32
bob,39
david,38
emelie,47
what i want to do is count how many names there are with different ages, so output would be like this ....
david,2
emelie,2
clarissa,1... (3 Replies)
Hi All,
I have a file which is having 3 columns as (string string integer)
a b 1
x y 2
p k 5
y y 4
.....
.....
Question:
I want get the unique value of column 2 in a sorted way(on column 2) and the sum of the 3rd column of the corresponding rows. e.g the above file should return the... (6 Replies)
Hi Friends,
I am having some trouble reading into an array. Basically, I am trying to grep for a pattern and extract it's value and store the same into an array. For eg., if my input is:
<L:RECORD>name=faisel farooq,age=21,
company=TCS,project=BT</L:RECORD>
<L:RECORD>name=abc... (1 Reply)