Solaris 10 v245 Busy/Slow System


 
Thread Tools Search this Thread
Operating Systems Solaris Solaris 10 v245 Busy/Slow System
# 1  
Old 07-27-2009
Question Solaris 10 v245 Busy/Slow System

Hello All,

I have noticed that one of my servers, the busiest has become increasingly slow to respond and execute commands, the running applications appear to be fine though.

Here is some output from vmstat :-

Code:
 kthr      memory            page            disk          faults      cpu
 r b w   swap   free  	re  mf 	 pi po 	 fr   de sr m0 m1 m2 m3   in   sy   cs  us sy id
 0 0 0  7141808 755736 	52  2  	 0  1188 768  0  0  0  0  0 157 1000 20150 2076 14 12 74
 0 0 0  7137720 755616 	237 1325 0  1514 1067 0  0  0  0  3 197 1260 30616 2558 20 20 60
 3 0 0  7141808 755712 	59  0    0  1321 860  0  0  0  0  4 172 1095 21384 2162 12 13 75
 10 0 0 7139320 754224 	432 2535 0  1624 1093 0  0  0  0  0 211 1478 33216 2796 23 23 53
 0 0 0  7141808 755792 	59  152  0  1513 1069 0  0  0  0  1 191 1172 24833 2250 15 17 68
 0 0 0  7136904 753064 	402 2047 0  1385 1011 0  0  79 0  0 189 1481 33519 2846 24 22 54
 0 0 0  7139416 755088 	302 949  0  1592 1091 0  0  10 0  0 210 1249 29910 2500 19 21 60
 0 0 0  7141816 755864 	195 1070 0  1931 1293 0  0  0  0  0 267 1565 34584 2834 21 20 59

I am trying to determine if this can be tuned out with system parameters or if the server requires additional memory.

Any ideas?


Wes

Last edited by DukeNuke2; 07-28-2009 at 03:45 AM..
# 2  
Old 07-28-2009
Whats the uptime of the server? If its really too long, you might want to consider getting a downtime to reboot (as an interim solution).
What Solaris OS is it running? prstat -a.. checkout the process that is consuming high CPu/Mem utilisation first
In my example below, you can see that the flarcreate command is the most resourse utilising.....

Code:
root@unknown # prstat -a
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 29535 root     2220K 1192K sleep    1    0   0:01:57  18% compress/1
 29489 root     1352K  744K sleep    1    0   0:01:06  10% wc/1
 29532 root       10M   10M sleep   60    0   0:00:12 1.6% cpio/1
 29534 root     1500K  736K sleep    1    0   0:00:09 1.5% computehash/1
 29533 root     1492K  720K sleep   24    0   0:00:04 0.7% computehash/1
 10698 root      136M   64M sleep   60    0   0:00:52 0.0% java/27
 11292 root      156M   84M sleep   60    0   0:02:13 0.0% java/30
 11589 noaccess  269M  115M sleep   31    0   0:00:54 0.0% java/26
 29569 siraj    7888K 3948K sleep   59    0   0:00:00 0.0% sshd/1
 12530 root      177M   68M sleep   18    0   0:00:20 0.0% java/37
  9991 root      208M  139M sleep   17    0   0:00:41 0.0% java/18
 29635 root     3628K 2828K cpu1    59    0   0:00:00 0.0% prstat/1
 29579 root     2952K 1728K sleep   59    0   0:00:00 0.0% bash/1
  9354 root       33M   25M sleep   59    0   0:00:06 0.0% Xorg/1
 29601 root     2948K 1732K sleep   59    0   0:00:00 0.0% bash/1
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU
    80 root      499M  547M   6.7%   0:08:19  32%
     1 noaccess  195M  172M   2.1%   0:00:54 0.0%
     5 siraj    2924K 7548K   0.1%   0:00:00 0.0%
     1 lp       1016K 1816K   0.0%   0:00:00 0.0%
     1 smmsp    1136K 4472K   0.1%   0:00:00 0.0%
Total: 97 processes, 419 lwps, load averages: 1.27, 0.57, 0.22
root@unknown # ptree 29535
8788  /usr/lib/ssh/sshd
  28547 /usr/lib/ssh/sshd
    28548 /usr/lib/ssh/sshd
      28554 -ksh
        28559 bash
          28561 -bash
            28583 bash
              29438 /bin/ksh /usr/sbin/flarcreate -n archive1 -c archive1.flar
                29487 /usr/bin/awk {print $1}
                  29488 /bin/ksh /usr/sbin/flarcreate -n archive1 -c archive1.f
                    29535 compress -fc


Last edited by incredible; 07-28-2009 at 03:29 AM..
# 3  
Old 07-28-2009
Thanks for the reply, the server has been up for 13 days so not very long.
Quote:
11:56am up 13 day(s), 11:35, 4 users, load average: 0.77, 0.72, 0.77
Its currently running Solaris 10 with the recommended updates, here is the showrev output.
Quote:
Hostname: **********
Hostid: **********
Release: 5.10
Kernel architecture: sun4u
Application architecture: sparc
Hardware provider: Sun_Microsystems
Domain:
Kernel version: SunOS 5.10 Generic_118833-36
The server uses very little CPU as most of our application work is done in memory. The server has 8GB of physical memory and approx 8GB swap available.

Quote:
NPROC USERNAME SIZE RSS MEMORY TIME CPU
462 blahblah 176G 169G 85% 88:35:59 13%
19 oracle 29G 29G 15% 4:10:03 0.9%
41 root 396M 124M 0.1% 3:15:24 0.5%
1 noaccess 193M 82M 0.0% 0:18:31 0.0%
1 smmsp 7632K 2016K 0.0% 0:00:01 0.0%
6 daemon 20M 11M 0.0% 0:00:02 0.0%


---------- Post updated at 01:02 PM ---------- Previous update was at 12:18 PM ----------

Here is the output from "prstat -s size -n 50", it looks like each oracle process is eating memory Smilie

Quote:
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1819 oracle 1678M 1642M sleep 59 0 2:14:28 0.3% oracle/14
1817 oracle 1673M 1648M sleep 59 0 0:08:45 0.0% oracle/258
1821 oracle 1668M 1642M sleep 59 0 0:07:38 0.0% oracle/15
1830 oracle 1667M 1648M sleep 59 0 0:03:03 0.0% oracle/1
1827 oracle 1667M 1643M sleep 59 0 0:17:03 0.0% oracle/1
1823 oracle 1667M 1643M sleep 59 0 0:01:25 0.0% oracle/1
1970 oracle 1666M 1646M sleep 59 0 0:02:49 0.0% oracle/1
1911 oracle 1666M 1647M sleep 59 0 0:00:20 0.0% oracle/1
3429 oracle 1666M 1645M sleep 59 0 0:43:20 0.1% oracle/1
2940 oracle 1666M 1645M sleep 59 0 0:00:41 0.0% oracle/1
3402 oracle 1666M 1645M sleep 59 0 0:00:58 0.0% oracle/1
1984 oracle 1666M 1646M sleep 59 0 0:12:37 0.1% oracle/1
1976 oracle 1666M 1646M sleep 59 0 0:09:39 0.0% oracle/1
1972 oracle 1666M 1646M sleep 59 0 0:11:04 0.0% oracle/1
1811 oracle 1666M 1642M sleep 59 0 0:14:16 0.0% oracle/1
1996 oracle 1666M 1645M sleep 59 0 0:05:43 0.0% oracle/1
1992 oracle 1666M 1645M sleep 59 0 0:09:57 0.0% oracle/1
25583 oracle 1666M 1648M sleep 59 0 0:00:03 0.0% oracle/1
1962 oracle 1666M 1646M sleep 59 0 0:06:13 0.0% oracle/1
1966 oracle 1666M 1645M sleep 59 0 0:00:26 0.2% oracle/1
2012 oracle 1666M 1642M sleep 59 0 0:00:08 0.0% oracle/1
1836 oracle 1666M 1640M sleep 59 0 0:00:03 0.0% oracle/1
2780 oracle 1666M 1646M sleep 59 0 0:00:43 0.0% oracle/1
17214 oracle 1666M 1645M sleep 59 0 0:01:36 0.0% oracle/1
1974 oracle 1666M 1646M sleep 59 0 0:11:19 0.0% oracle/1
3609 oracle 1666M 1643M sleep 59 0 0:00:00 0.3% oracle/1
4533 oracle 1666M 1645M sleep 59 0 0:05:43 0.0% oracle/1
2916 oracle 1666M 1645M sleep 59 0 0:01:30 0.0% oracle/1
20923 oracle 1666M 1644M sleep 59 0 0:00:00 0.0% oracle/1
1954 oracle 1666M 1646M sleep 59 0 0:06:49 0.0% oracle/1
2886 oracle 1666M 1645M sleep 59 0 0:03:45 0.0% oracle/1
1834 oracle 1666M 1636M sleep 59 0 0:00:01 0.0% oracle/1
1988 oracle 1665M 1645M sleep 59 0 0:07:16 0.0% oracle/1
1986 oracle 1665M 1646M sleep 59 0 0:06:20 0.0% oracle/1
1980 oracle 1665M 1646M sleep 59 0 0:08:47 0.0% oracle/1
1978 oracle 1665M 1645M sleep 59 0 0:11:27 0.0% oracle/1
1968 oracle 1665M 1646M sleep 59 0 0:03:45 0.0% oracle/1
1964 oracle 1665M 1646M sleep 59 0 0:04:39 0.1% oracle/1
1960 oracle 1665M 1646M sleep 59 0 0:08:03 0.0% oracle/1
1958 oracle 1665M 1645M sleep 59 0 0:03:53 0.0% oracle/1
3421 oracle 1665M 1645M sleep 59 0 0:06:24 0.0% oracle/1
2910 oracle 1665M 1645M sleep 59 0 0:01:17 0.0% oracle/1
2892 oracle 1665M 1645M sleep 59 0 0:03:40 0.0% oracle/1
1994 oracle 1665M 1645M sleep 59 0 0:10:50 0.0% oracle/1
1982 oracle 1665M 1646M sleep 59 0 0:06:14 0.0% oracle/1
14376 oracle 1665M 1645M sleep 59 0 0:07:52 0.0% oracle/1
1990 oracle 1665M 1645M sleep 59 0 0:06:27 0.0% oracle/1
1956 oracle 1665M 1646M sleep 59 0 0:03:02 0.0% oracle/1
3906 oracle 1665M 1644M sleep 59 0 0:00:08 0.0% oracle/1
754 oracle 1665M 1645M sleep 59 0 0:01:41 0.0% oracle/1
Total: 529 processes, 2429 lwps, load averages: 0.61, 0.70, 0.77
# 4  
Old 07-28-2009
try the ps command to sort out using memory. Based on this we can find which oracle process is consuming more memory and what tasks are going on using these processes

ps -eo pmem,pcpu,rss,vsz,args | sort -k 1 -r | more

Regards
Ramkumar
# 5  
Old 07-28-2009
By the way, Generic_118833-36 is far too old. Has so many bug fixes with this release. Please update the patch anyway,
# 6  
Old 07-28-2009
I think we may have found the issue, our Oracle 10 installations were completed without adding an oracle user project to define memory usage. We have since learned that without this it will take roots default values which are listed below.

Maybe 16.0EB is too much Smilie

Quote:
bash-3.00# prctl -i project user.oracle
prctl: user.oracle: cannot find project

bash-3.00# prctl -i project user.root

project: 1: user.root
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-contracts
privileged 10.0K - deny -
system 2.15G max deny -
project.max-device-locked-memory
privileged 500MB - deny -
system 16.0EB max deny -
project.max-port-ids
privileged 8.19K - deny -
system 65.5K max deny -
project.max-shm-memory
privileged 1.96GB - deny -
system 16.0EB max deny -
project.max-shm-ids
privileged 128 - deny -
system 16.8M max deny -
project.max-msg-ids
privileged 128 - deny -
system 16.8M max deny -
project.max-sem-ids
privileged 128 - deny -
system 16.8M max deny -
project.max-crypto-memory
privileged 1.96GB - deny -
system 16.0EB max deny -
project.max-tasks
system 2.15G max deny -
project.max-lwps
system 2.15G max deny -
project.cpu-shares
privileged 1 - none -
system 65.5K max none -
zone.max-lwps
system 2.15G max deny -
zone.cpu-shares
privileged 1 - none -


---------- Post updated at 05:42 PM ---------- Previous update was at 05:41 PM ----------

Quote:
Originally Posted by incredible
By the way, Generic_118833-36 is far too old. Has so many bug fixes with this release. Please update the patch anyway,
I am looking on the SUN Software site for the latest patch release.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Slow UNIX system this morning

Hi, what would be the first things to check on a system that normally works fine, and is not so fine this morning ? Its accessing menus and various other screens 100x slower than normal. Version: UnixWare 5 7.1.3 i386 SCO UNIX_SVR5 I have tried this pf -ef|grep paulc and found a huge list of... (6 Replies)
Discussion started by: Mick_Dundee
6 Replies

2. Red Hat

GFS file system performance is very slow

My code Hi All, I am having redhat linux 5.3 (Tikanga) with GFS file system and its very very slow for executing ls -ls command also.Please see the below for 2minits 12 second takes. Please help me to fix the issue. $ sudo time ls -la BadFiles |wc -l 0.01user 0.26system... (3 Replies)
Discussion started by: susindram
3 Replies

3. UNIX for Advanced & Expert Users

slow performing system

Friends Greetings. I have a RedHat 5.7 64bit virtual server on VMware ESXi 4.1. This server and other Redhat Servers are running very slow. I did some stats collection on ESXi and looks like Linux is holding the disk IO. I am not sure what is causing this behavior. On Linux I checked the CPU... (4 Replies)
Discussion started by: sdewal
4 Replies

4. Solaris

Finding most busy file system

Hi Experts, I was asked to find most busy file system on one of the server. It is Sun 10. Any idea to get this? Thanks, Deepak (5 Replies)
Discussion started by: naw_deepak
5 Replies

5. UNIX for Advanced & Expert Users

Filesystem mystery: disks are not busy on one machine, very busy on a similar box

Hi, We have a filesystem mystery on our hands. Given: 2 machines, A and Aa. Machine Aa is the problem machine. Machine A is running Ubuntu, kernel 2.6.22.9 #1 SMP Wed Feb 20 08:46:16 CST 2008 x86_64 GNU/Linux. Machine Aa is running RHEL5.3, kernel 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38... (2 Replies)
Discussion started by: mschwage
2 Replies

6. Solaris

SunFire V245, External CD/DVD-ROM, Solaris Install

Hey all. Wondering - I have a pair of V245's with no internal CD/DVD-ROM drives, and I want to install the latest/greatest Solaris 10. Booting these machines with an external drive plugged in via USB does no good. Is jumpstart or installing internal drives the only alternative? I'm a... (1 Reply)
Discussion started by: b1f30
1 Replies

7. Solaris

how to go to system console through ALOM in Sunfire V245

Hello , I am connecting sumfire v245 server to my laptop Through ALOM method.i got the ALOM prompt but after that i am typing sc> console to go to system console but i am not able to,after typing console only this line is coming Type #. to go back to ALOM. What to do ?? That server having... (5 Replies)
Discussion started by: avi157
5 Replies

8. HP-UX

What is lvmkd and why does it slow my system down?

I am running HPUX 11.0 with HP MirrorDisk/UX and recently had a HDD failure and replaced the drive with a new one. I did a "dd" command to copy the new data to the new drive and now I have a bunch of "lvmkd" in my "ps -ef" output. These processes are gobbling up CPU time and slowing my system... (2 Replies)
Discussion started by: jpetrecca
2 Replies

9. UNIX for Dummies Questions & Answers

Slow System

Hi, I have an SCO-Unix server running. There are some processes (unknown to me) which consume a lot of the system resources. This slows down the server dramatically. Is there a command or program that monitors what processes are using the cpu, disk, etc.. and tell me how excessive and how... (3 Replies)
Discussion started by: Hansaplast
3 Replies
Login or Register to Ask a Question