Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Count number of unique values in each column of array

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Count number of unique values in each column of array
# 15  
Old 01-16-2018
You have given us a 57.9KB a.txt and a 575.5KB a1.txt (which is created by your script from a.txt and should, therefore, be smaller than a.txt but is instead almost 10 times larger). Your script also creates a2.txt from a1.vcf. But, you haven't shown us what the contents of a1.vcf look like.

Please show us:
  1. the a1.txt that should be created from the sample a.txt you provided in post #14,
  2. a sample a.vcf file and a description of its contents (explaining what the field separator is in this file, what fields are used from which lines), and
  3. the exact output you hope to produce from those sample a.txt and a.vcf files.
This User Gave Thanks to Don Cragun For This Post:
Geneanalyst
# 16  
Old 01-16-2018
Quote:
Originally Posted by Don Cragun
You have given us a 57.9KB a.txt and a 575.5KB a1.txt (which is created by your script from a.txt and should, therefore, be smaller than a.txt but is instead almost 10 times larger). Your script also creates a2.txt from a1.vcf. But, you haven't shown us what the contents of a1.vcf look like.

Please show us:
  1. the a1.txt that should be created from the sample a.txt you provided in post #14,
  2. a sample a.vcf file and a description of its contents (explaining what the field separator is in this file, what fields are used from which lines), and
  3. the exact output you hope to produce from those sample a.txt and a.vcf files.
Hi Don, a1.vcf is a typo. It should be a1.txt. For some reason, I don't have an edit button for post 14.

I clipped off the bottom of a.txt because the file was a large 86M file. The bottom part is not necessary because it is all a bunch of 0/0 0/1 1/1 and ./.. and a repeat of the preceding rows I just wanted to show the header part plus some of the data (0/0 0/1 1/1 ./.)

The last portion of post 14 shows the desired output, with a column for the sample names ( row 28 in a.txt), the counts of values for each sample; 0/0, 0/1, 1/1, ./. (from a2.txt), a column for the SUM of 0/0 and 0/1 values. A sorting from high to low by column containing the SUM of 0/0 and 0/1.
# 17  
Old 01-22-2018
Quote:
Originally Posted by Geneanalyst
Thanks Don. Here is the my whole code:

Code:
# Identify target derived alleles to the exculsion of outgroups
awk -F "\t" '(NR>28) { if(($313 == "0/0") && ($314 != "0/0") && ($315 != "0/0") && ($316 != "0/0") && ($317 != "0/0") && ($318 != "0/0") && ($319 != "0/0")) {print $0} }' a.txt > a1.txt
#
# Strip columns 1-9 and write to a2.txt
awk '{for(i=10;i<=NF;i++){printf "%s ", $i}; printf "\n"}' a1.vcf > a2.txt
# PRINT HEADER
awk 'FNR==28 {for(i=9;i<=NF;i++){printf "%s ", $i}; printf "\n"}' a.txt
# Print count of 0/0 0/1 1/1 ./. for each sample in the run
awk '
{ 
   mc = NF > mc ? NF : mc
   for(i=NF; i; i--) {
      T[$i]
      C[i FS $i]++
   }
}
END {
  for(v in T) {
     printf "\n%s", v
     for(i=1; i<=mc;i++) printf "\t%d",C[i FS v]
  }
  printf "\n"
}' a2.txt

In the 1st part, the input data a.txt (attached file. I only copied the 1st 100 rows to include the header which has the sample names) is queried. Columns 1-9 do not contain relevant information. Columns 313-319 contain the target samples against which all the test samples are compared ( columns 10-312).

The rows that survive the comparison operation are written to a1.txt ( columns 1-9 don't contain relevant information).

Next the header containing the sample names is extracted from a.txt, and your code is executed for counting the number of unique values.

Next I manually add the values in the 0/1 and 1/1 columns, and create a totals column. I then sort the total column from high to low. The sample with the highest total indicates the most similarity to target sample (column 313).

I like the way you transposed the result and would like to also transpose the header with sample names. So instead of COL1. COL2, ..., I would like the sample names from row 28 ( columns 10-319), such as shown below.

Code:
FORMAT    1/1    0/0    0/1    TOTAL 0/0 & 0/1
.Kurd_C3_ID001    78    183    201    384
Balochi_HGDP00052    86    175    201    376
Balochi_HGDP00054    71    166    225    391
Balochi_HGDP00056    71    158    233    391
Balochi_HGDP00058    90    168    204    372
Balochi_HGDP00062    91    148    223    371
Balochi_HGDP00064    85    183    194    377
Balochi_HGDP00066    79    185    198    383
Balochi_HGDP00068    95    163    202    365
Balochi_HGDP00072    75    168    217    385
Balochi_HGDP00074    80    198    183    381
Balochi_HGDP00078    89    171    199    370
Balochi_HGDP00080    88    149    222    371
Balochi_HGDP00082    85    179    195    374
Balochi_HGDP00086    102    162    198    360
Balochi_HGDP00088    89    175    194    369
Balochi_HGDP00090    87    177    197    374
Balochi_HGDP00092    87    191    184    375
Balochi_HGDP00096    87    166    207    373
Balochi_HGDP00098    95    190    175    365
GujaratiD_NA20847    74    168    220    388
GujaratiD_NA20899    86    183    193    376

Moderator's Comments:
Mod Comment Please use CODE tags around sample input and output as well as around code segments.
I note that this output doesn't include any output for the fields that have the value ./.. Do you only want to display data in your output for the 1/1, 0/0, and 0/1 value counts?

Does the output order matter for the middle three columns?

You also said that your output should be sorted in decreasing order on the values in the last column, but your sample output appears to be unsorted???

Are columns 313-319 supposed to be counted and printed along with the test samples, or are just columns 10-312 supposed to be counted and printed?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Count unique column

Hello, I am trying to count unique rows in my file based on 4 columns (2-5) and to output its frequency in a sixth column. My file is tab delimited My input file looks like this: Colum1 Colum2 Colum3 Colum4 Coulmn5 1.1 100 100 a b 1.1 100 100 a c 1.2 200 205 a d 1.3 300 301 a y 1.3 300... (6 Replies)
Discussion started by: nans
6 Replies

2. UNIX for Beginners Questions & Answers

Awk: count unique element of array

Hi, tab-separated input: blabla_1 A,B,C,C blabla_2 A,E,G blabla_3 R,Q,A,B,C,R,Q output: blabla_1 3 blabla_2 3 blabla_3 5 After splitting $2 in an array, I am trying to store the number of unique elements in a variable, but have some difficulties resetting the variable to 0 before... (6 Replies)
Discussion started by: beca123456
6 Replies

3. Shell Programming and Scripting

Print count of unique values

Hello experts, I am converting a number into its binary output as : read n echo "obase=2;$n" | bc I wish to count the maximum continuous occurrences of the digit 1. Example : 1. The binary equivalent of 5 = 101. Hence the output must be 1. 2. The binary... (3 Replies)
Discussion started by: H squared
3 Replies

4. Shell Programming and Scripting

Count occurrence of column one unique value having unique second column value

Hello Team, I need your help on the following: My input file a.txt is as below: 3330690|373846|108471 3330690|373846|108471 0640829|459725|100001 0640829|459725|100001 3330690|373847|108471 Here row 1 and row 2 of column 1 are identical but corresponding column 2 value are... (4 Replies)
Discussion started by: angshuman
4 Replies

5. Shell Programming and Scripting

Count frequency of unique values in specific column

Hi, I have tab-deliminated data similar to the following: dot is-big 2 dot is-round 3 dot is-gray 4 cat is-big 3 hot in-summer 5 I want to count the frequency of each individual "unique" value in the 1st column. Thus, the desired output would be as follows: dot 3 cat 1 hot 1 is... (5 Replies)
Discussion started by: owwow14
5 Replies

6. UNIX for Dummies Questions & Answers

count number of distinct values in each column with awk

Hi ! input: A|B|C|D A|F|C|E A|B|I|C A|T|I|B As the title of the thread says, I would need to get: 1|3|2|4 I tried different variants of this command, but I don't manage to obtain what I need: gawk 'BEGIN{FS=OFS="|"}{for(i=1; i<=NF; i++) a++} END {for (b in a) print b}' input ... (2 Replies)
Discussion started by: beca123456
2 Replies

7. Shell Programming and Scripting

How to count Unique Values from a file.

Hi I have the following info in a file - <Cell id="25D"/> <Cell id="26A"/> <Cell id="26B"/> <Cell id="26C"/> <Cell id="27A"/> <Cell id="27B"/> <Cell id="27C"/> <Cell id="28A"/> I would like to know how would you go about counting all... (4 Replies)
Discussion started by: Prega
4 Replies

8. UNIX for Dummies Questions & Answers

count number of rows based on other column values

Could anybody help with this? I have input below ..... david,39 david,39 emelie,40 clarissa,22 bob,42 bob,42 tim,32 bob,39 david,38 emelie,47 what i want to do is count how many names there are with different ages, so output would be like this .... david,2 emelie,2 clarissa,1... (3 Replies)
Discussion started by: itsme999
3 Replies

9. Shell Programming and Scripting

print unique values of a column and sum up the corresponding values in next column

Hi All, I have a file which is having 3 columns as (string string integer) a b 1 x y 2 p k 5 y y 4 ..... ..... Question: I want get the unique value of column 2 in a sorted way(on column 2) and the sum of the 3rd column of the corresponding rows. e.g the above file should return the... (6 Replies)
Discussion started by: amigarus
6 Replies

10. Shell Programming and Scripting

Not able to read unique values in array

Hi Friends, I am having some trouble reading into an array. Basically, I am trying to grep for a pattern and extract it's value and store the same into an array. For eg., if my input is: <L:RECORD>name=faisel farooq,age=21, company=TCS,project=BT</L:RECORD> <L:RECORD>name=abc... (1 Reply)
Discussion started by: faiz1985
1 Replies
Login or Register to Ask a Question