awk based script to print the "mode(statistics term)" for each column in a data file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk based script to print the "mode(statistics term)" for each column in a data file
# 1  
Old 06-26-2012
awk based script to print the "mode(statistics term)" for each column in a data file

Hi All,
Thanks all for the continued support so far.
Today, I need to find the most occurring string/number(also called mode in statistics terminology) for each column in a data file (.csv type).
For one column of data(1.txt) like below
Code:
Sample
1
2
2
3
4
1
1
1
2

I can find the mode using the command
Code:
sort 1.txt |uniq -c |sort |tail -1 |awk '{print $2}'

. The output is 1. ( As '1' appeared more number of times than other numbers)
But if I have multiple columns in a csv file, I am unable to do this task for each column printing the mode side by side. So I need an awk based loop script to fulfill my requirement.
Please let me know if my problem statement is not clear.

Thanks
Sid
# 2  
Old 06-26-2012
Bug try this

Code:
cat test_st
1 2 3
2 4 5
2 3 5
3 4 9
4 7 5
1 5 6
1 4 5
1 4 2
2 5 5

script is
Code:
for i in {1..3}
do
echo -e "i == $i\n"
cat test_st | awk -v L=$i '{ print $L }' | sort | uniq -c | sort -nrk1 | head -1 | awk '{ print $2 }'
done

output is
Code:
i == 1

1
i == 2

4
i == 3

5

# 3  
Old 06-26-2012
This script is perfect for files with known number of columns

Hi Pamu,

Thanks for your quick reply.
This script is perfectly fine for all files which have equal number of columns.
My challenge is to find 'mode' for files(.csv type) where
1) I don't know how many columns are there unless I open each file and check(very tedious task)
2) Printing the output side by side like mode_of_col1,mode_of_col2 etc.
For the first one i can count the number of columns in each file and feed into variable and loop through.
For the second one I can use the command
Code:
tr '\n' ' '

to transpose the output.

But all these change are making my code very lengthy and also I need to process many files (some times in 1000's).
Thats why I am looking for automatic looping script to print 'mode' for each column as well as looping to random number of columns.
Can you modify your script if you can ?

Thanks
Sid
# 4  
Old 06-26-2012
Try...
Code:
$ cat 1.txt
Sample
1
2
2
3
4
1
1
1
2

$ awk '{a[$1]++}END{for(i in a)if(a[i]>x)x=a[i];for(i in a)if(a[i]==x)print i}' 1.txt
1

$

Note that if the mode is not unique, more than one value is returned...
Code:
$ echo 2 >> 1.txt

$ awk '{a[$1]++}END{for(i in a)if(a[i]>x)x=a[i];for(i in a)if(a[i]==x)print i}' 1.txt
1
2

$

# 5  
Old 06-26-2012
Hi ks_reddy,

you can do two things here..

1) for getting number of columns you can get the number columns from the file and use this as variable as you have suggested.

2) For getting the output in one line better you redirect the output to one file and then you can perform any operation on it.

3) For many number of files just do this one while loop and redirect the output the file which name is appended by the original file.

and length of code doesn't matter too much if we are getting our work done.. Smilie
# 6  
Old 06-26-2012
Try...
Code:
$ cat 1.txt
5,3,2,6
4,4,3,2
4,1,2,3
4,1,2,3
3,5,2,3

$ awk ' BEGIN { FS=OFS="," }
      { for (n = 1; n <= NF; n++) { a[n, $n]++ } }
  END { for (n = 1; n <= NF; n++) {
          x = 0
          for (i in a) {
            split(i, s, SUBSEP)
            if (s[1] == n && a[i] > x) {
              x = a[i]
              y = s[2]
            }
          }
          printf y (n==NF ? ORS : OFS)
        }
      } ' 1.txt > 2.txt

$ cat 2.txt
4,1,2,3

$

Note that if the mode is not unique, only one value is returned.
# 7  
Old 06-26-2012
This script is perfect for my requirement.

Hi Ygor,

Thank you very much. You provided me the exact solution that I am looking for .
This script is taking only 1/10000 part of time that the previous script (which works on individual columns ).

Thanks
Sidda
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

2. Shell Programming and Scripting

Why awk print is strange when I set FS = " " instead of FS = "\t"?

Look at the following data file(cou.data) which has four fields separated by tab. Four fields are country name, land area, population, continent where it belongs. As for country name or continent name which has two words, two words are separated by space. (Data are not accurately... (1 Reply)
Discussion started by: chihuyu
1 Replies

3. Shell Programming and Scripting

replace the contents of first column of file "X" with second Column of file "X" in file "Y"

Hi! I am having 02 files. In first file" X" I am having 02 Columns TCP-5100 Sybase_5100 TCP-5600 Sybase_5600 Second file "Y" for example-- :services ( :AdminInfo ( :chkpf_uid ("{A2F79713-B67D-4409-83A4-A90804E983E9}") :ClassName (rule_services) ) :compound ()... (12 Replies)
Discussion started by: shahid1632
12 Replies

4. UNIX for Dummies Questions & Answers

Replacing "." with "GG" in a certain column of a file that has heading

Hi, all, I have a file that looks like: ## XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ## YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY #AA AB AC AD AE AF AG AH AI AJ AK AL 20 60039 60039 ... (5 Replies)
Discussion started by: kush
5 Replies

5. Shell Programming and Scripting

Choosing between repeated entries based on the "absolute values" of a column

Hello, I was looking for a way to select between the repeated entries (column1) based on the values of absolute values of column 3 (larger value). For example if the same gene id has FC value -2 and 1, I should get the output as -2. Kindly help. GeneID Description FC ... (2 Replies)
Discussion started by: Sanchari
2 Replies

6. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

7. Shell Programming and Scripting

Simplify Bash Script Using "sed" Or "awk"

Input file: 2 aux003.net3.com error12 6 awn0117.net1.com error13 84 aux008 error14 29 aux001.ha.ux.isd.com error12 209 aux002.vm.ux.isd.com error34 21 alx0027.vm.net2.com error12 227 dux001.net5.com error123 22 us008.dot.net2.com error121 13 us009.net2.com error129Expected Output: 2... (4 Replies)
Discussion started by: sQew
4 Replies

8. Shell Programming and Scripting

"Join" or "Merge" more than 2 files into single output based on common key (column)

Hi All, I have working (Perl) code to combine 2 input files into a single output file using the join function that works to a point, but has the following limitations: 1. I am restrained to 2 input files only. 2. Only the "matched" fields are written out to the "matched" output file and... (1 Reply)
Discussion started by: Katabatic
1 Replies

9. Shell Programming and Scripting

cat $como_file | awk /^~/'{print $1","$2","$3","$4}' | sed -e 's/~//g'

hi All, cat file_name | awk /^~/'{print $1","$2","$3","$4}' | sed -e 's/~//g' Can this be done by using sed or awk alone (4 Replies)
Discussion started by: harshakusam
4 Replies

10. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies
Login or Register to Ask a Question