Originally Posted by bakunin

I will elaborate further on this later, just a few impressions on first glance:

Your system seems to have not enough RAM for what you are doing. The result is some heavy swapping going on. The vmstat output is somewhat different from the system i usually work on (AIX), but a few things never change:

Look at the first two columns, named "r" and "b". The "r" is the number of running processes at that time, "b" is the number of blocked processes. "blocked" means the process would be ready to run but has to wait for some reason. In a majority of cases this is because it was swapped out earlier and now is waiting until swapping it in has finished.

The depth of the blocked-queue should therefore ideally be a constant zero - everything else is alarming.

On the right part of the output the CPU activity is shown. "us" "sy" "id" "wa" are percent values and add up to 100. They show how much time the CPU has spent in user space, system routines, idling and waiting. In an ideal world the CPU would spent most of its time in us and sy, the rest would go to id and wa would be 0. Every time the wa value is bigger than 0 this means that the CPU has found nothing productive to do. Most of the times this is also a side effect of processes swapping in and out, because as long as the swapping goes on the CPU can do nothing but wait, save for the few system calls necessary for the swapping itself.

Ok, I'm in a bit of a hurry, later more.

bakunin