UNIX for Advanced & Expert Users

hi,
i have 10 processes running on a tru64
machine with 16 cpu. when i am checking the states of these processes
with ps command, i saw that almost never more than 3 of them are in the
R(unning) state. (generally they are in the S(leeping) state).

although there are many other processes running on the same machine,
my expectation is to see most of my processes in the running state,
because top reports generally we have 20%-50% idle cpu,

even if they are waiting for some resource (e.g: io), they should be
in U(ninterruptable) state, am i wrong?

how can this sleeping state problem can be explained?

If a process is waiting for a resource it will almost always be asleep and will be woken by the kernel when the resource is available. CPU's do not simply run processes...it might be running kernel code to make those needed resources available. An idle CPU is looping repeatedly scanning the run queue looking for something to do... no other state is counted as idle.

The exception to sleeping for a resource is spinlocking...just looping until the resource is free (because the resource is expected to be locked very briefly). This happens in the kernel in an SMP environment. 16 cpus is a lot and you probably spend a fair amount of time spinlocked. Spinlocking if why you can't have unlimited numbers of cpu's on an SMP box...you reach a point where you spinlock too much.

S(leeping) state means that process is waiting for a resource.
then, is it possible to determine what is that resource?
process is waiting for what?
using lsof, iostat,netstat,vmstat or something else on tru64?

thanks in advance

Well, I have never used Tru64, so I'm not sure how hard that will be. But at some level the answer is yes, it is possible. A sleeping process will have a value usually called a "wait channel" that relates to what it is waiting for. ps often can display this field labelled "WCHAN" or something like that. It is a address in the kernel's memory space. Sometimes you can figure it out with adb. Some recent BSD versions actually decode this field. HP has a product called Glance that can decode it for HP-UX. I know that HP has a version of Glance for Solaris. I don't know if they have one for Tru64 or not. If they do, it is worth getting. It is the best performance tool I have seen.