Shell Programming and Scripting

Hi All,

I'm confused what to use (Python/Perl/Shell), I have a scenario here wherein i need to process a data which might have millions of records in tabular format, my task is to find the {0,null,NA} in each and every column and also inform the end user that this column has this many values accumulated.

for ex :-

Code:

Code:

id	name 	desc
1	Raj	null
2	Nick	NA
0	Digu	fine with it
3	NA	go ahead
4	Duvi	baby

So in above table id has only 0 occuring only once so the ratio would be 80%, similarly name has NA only once it also has ratio of 80% and desc column has both (NA,null). So the ratio would be 60%.

O/P should tell the name of the column along with the ratio.

PS : The data may be millions of records, so we are concerned about time complexity

not sure. try:

Code:

awk  '
NR==1 {for (cc=1; cc<=NF; cc++) n[$cc]=$cc; t=$0; next;}
{
   if ($1 != "0") c[1]++;
   for (i=2; i<=NF; i++) if ($i != "NA" && $i != "null" ) c[i]++;
}
END {
   print t;
   --NR
   r="";
   for (i=1 ; i<cc; i++) {
      p=(c[i]/NR)*100;
      r=(i == 1) ? "" p : r OFS p;
   }
   print r
}
' FS="\t" OFS="\t" infile

thanks for the solution, but i forgot to mention that my delimiter here is "|" symbol.
How do i proceed?

I got the solution, just changed the

Code:

Code:

FS="|" OFS="|" infile