Shell Programming and Scripting

1. I can not post an example with 1 million lines on here. The data is exactly the same in terms of style of content, it just goes on for another 999.997 lines.
2. I stated in the first post that writing NR>1 gave me the last number of the column. I'm sorry I didn't realize that in my example this number was in fact the smallest one, but in my original file it clearly is not, that I can see from using the head-command. Using tail, I accidentally discovered that it was the last entry in the column that was posted instead.
3. As stated, both scripts work. I used it on a data set I have of about 300 rows, which I'm able to check using excel, something I can't do with the largest data set, as excel will most likely crash.

Have you tried my solution.. ?

Please check below..

Code:

$ cat file
   FID        IID MISS_PHENO   N_MISS   N_GENO   F_MISS
  12AB43131   12AB43131          N    17774   906341  0.01961
  65HJ87451   65HJ87451          N    10149   906341   0.0112
  43JJ21345   43JJ21345          N     2826   906341 0.003118
  43JJ21345   43JJ21345          N     2826   906341 0.3119
  43JJ21345   43JJ21345          N     2826   906341 0.3118

$ awk 'NR>1{a+=$NF;
  max=max>$NF?max:$NF;
  min=min>$NF||!min?$NF:min}
  END{print a/(NR-1),max,min}' file
0.131526 0.3119 0.003118

Same result got using Rudic's script.

Yes, pamu, yours worked as well. I tested both on the 300 row file, and they gave the exact same output, and was verified by excel.

I hope to be able to understand more of the content of your script some day, but right now I have a deadline, so I will have to get back to it later. Looks really interesting though!

I did some minor changes to RudiC's script to fix line breaks etc. Is that what you were referring to, pamu?

Ya Sure.

You can use both the script to get your desired output.

And i think you can use awk for large records. Unlike excel it won't crash.. (probably)

Yes, this is why I posted the question in the first place. I did my entire pipeline of analysis (including much more than just this) using excel to do editing etc, but it just kept crashing and saving just a fraction of the file etc.

Redoing with awk now, which runs really smoothly in comparison, just need to get all the scripts right.

Thank you so much for all your help!