Shell Programming and Scripting

So I have several files (35000, to be exact) in the format rmsd_protein_*.dat each with 2 columns and 35000 rows.

I would like to count how many values in the second column are less than 3 for each file, and output it into a new file so that it ultimately appears as:

Code:

1           14057
2           20598

.....

where the left column is the * value from the data file, and the number on the left is the number of values from that file less than three.

Unfortunately, I'm a n00b as far as shell scripts go, and even though this would be the easiest thing in the world in JAVA I just can't figure it out in scripts. Thoughts?

Code:

#!/bin/sh

ls -1 | grep "rmsd_protein_.*.dat" | while read file
do
   f=${file##*_}
   f=${f%%[.]*}
   awk '$2 < 3 {c++} END {print f, c}' f=$f $file
done > new_file.dat

Unfortunately I get the error "while: Expression Syntax." when I try to run that...?

try sh, bash, or ksh.

C shell (on Red Hat Enterprise Linux Workstation release 6.5 (Santiago), if that matters in any way)

for csh:

Code:

#!/bin/csh

foreach file ( "`ls -1 | grep 'rmsd_protein_.*.dat' `" )
   awk '$2 < 3 {c++} END {sub(".*_", "", f); sub("[.].*", "", f); print f, c}' f=$file $file
end

I am unsure what you mean exactly, but I take it you want one line for each file with the number of lines where $2 was <3 plus the total number of lines for that file.. If so, this should be fairy quick:

Code:

find . -name 'rmsd_protein_*.dat' -exec awk 'FNR==1{if(NR>1) print c,t; c=0} $2<3{c++}{t=FNR} END{print c,t}' OFS='\t' {} + > newfile.dat