Solaris

Good morning,
I need some help figuring out what's eating up my cpu. My application can't get enough cpu to do its job. this is a sunfire V440 2CPU's at 1/593 GHZ with 8GB of memory. In the morning hours the box is at less than 3%. I can't figure out what else is using the CPU. We use foglight and it shows that my user is not using much at all.


look at it now:

load averages: 5.62, 3.96, 3.60 us352 08:46:48
164 processes: 159 sleeping, 3 running, 2 on cpu
CPU states: 0.5% idle, 12.9% user, 86.6% kernel, 0.0% iowait, 0.0% swap
Memory: 8.0G real, 5.3G free, 511M swap in use, 13.1G swap free

PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
9196 ceunix 1 10 0 4048K 2496K sleep 0:36 30.19% cmumboxd
5917 ceunix 1 39 0 6336K 4448K run 0:03 2.54% cmusvwd
27764 ceunix 1 49 0 5312K 2168K sleep 19:13 2.19% cmuctld
9784 ceunix 1 50 0 4048K 2504K sleep 0:00 1.50% cmumboxd
9800 ceunix 1 59 0 4080K 2656K sleep 0:00 0.84% cmumboxd
9797 ceunix 1 50 0 4048K 2344K sleep 0:00 0.78% cmumboxd
9804 ceunix 1 0 0 4032K 1464K sleep 0:00 0.60% cmumboxd
8600 ceunix 1 59 0 7568K 3216K run 0:01 0.56% sshd
27787 ceunix 1 59 0 7528K 6128K sleep 0:43 0.34% cmuacd
27765 ceunix 1 59 0 5328K 2384K sleep 2:01 0.26% cmuauthd
9781 ceunix 1 59 0 4072K 2536K sleep 0:00 0.26% cmumboxd
9303 ceunix 1 59 0 4080K 2600K sleep 0:00 0.19% cmumboxd
27785 ceunix 1 23 0 4040K 2504K sleep 1:40 0.18% cmumboxd
8617 ceunix 1 59 0 4080K 2560K sleep 0:00 0.14% cmumboxd
15234 ceunix 1 49 0 6816K 4000K cpu/0 0:06 0.13% top


kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m1 m3 m1 in sy cs us sy id
1 0 0 14402544 6435424 1249 2043 3006 1 1 0 1 13 0 0 8 1351 17468 1095 16 28 56

From the output given, it shows that

cmumboxd
is utilizing the highest CPU and 30+% is a lot. Maybe you should renice the process?

Also did u check vmstat? I dont think that it is really swapping much as it seems a huge amount of RAM is free.
cmumboxd is making too many system calls and utilizing CPU heavily. Signs of bad coding eh?

~Sage

We found the culprit. It was one of our clean up scripts. We installed Dtrace on the solaris kernel which helped us track this script that was basically doing some intense i/o calls. thanks so much for replying to my post.