Poor Performance of server

guzzelle · #1 02-07-2012

Hi,

I am new registered user here in this UNIX forums.
I am a new system administrator for AIX 6.1. One of our servers performs poorly every time our application (FINACLE) runs many processes/instances. (see below for topas snapshot)

I use NMON or Topas to monitor the server utilization. I checked the the CPU Idle% and the idle percent is high, however the DISK Busy% is constantly high (during real poor performance, the DISK Busy% is most of the time 100%). Also, I noticed that the FILE/TTY Readch and Writech are constantly high too. See topas snapshot below:

Code:

CPU User% Kern% Wait% Idle% Physc Entc
ALL 0.7 0.4 5.5 93.5 0.10 1.7
 
Disk Busy% KBPS TPS KB-Read KB-Writ
Total 100.0 10.8K 219.0 0.0 10.8K
 
FileSystem KBPS TPS KB-Read KB-Writ
Total 6.5K 648.6 3.8K 2.7K
 
Name PID CPU% PgSp Owner
oracle 1311046 0.6 10.6 oracle
vmmd 458766 0.2 1.2 root
aioserve 10682432 0.0 0.4 uatadm2
topas 25755848 0.0 8.9 bankadm
tnslsnr 25493548 0.0 20.4 oracle
oracle 5439982 0.0 14.7 oracle
 
EVENTS/QUEUES FILE/TTY
Cswitch 585 Readch 3915.9K
Syscall 2055 Writech 2759.0K
Reads 630 Rawin 0
Writes 90 Ttyout 1628
Forks 1 Igets 0
Execs 0 Namei 99
Runqueue 1.1 Dirblk 0
Waitqueue 0.5
Memory
PAGING Real,MB 43776
Faults 1639 % Comp 47
Steals 0 % Noncomp 51
PgspIn 0 % Client 51
PgspOut 0
PageIn 0 PAGING SPACE
PageOut 691 Size,MB 12288
Sios 700 % Used 1
% Free 99
NFS (calls/sec)
SerV2 0 WPAR Activ 0
CliV2 0 WPAR Total 0
SerV3 0 Press: "h"-help
CliV3 0 "q"-quit

zaxxon · #2 02-08-2012

Killing processes to free resources is not a good idea. You might shoot something you still need.

Yes, from the look of it you have a severe bottleneck with your 1 hdisk. Is this hdisk a physical disk or a LUN from SAN storage?
Do you use asynchronous I/O (AIO) and have it tuned? Oracle will most probably benefit from it as well as getting additional disks.

nmon/topaz has a page that displays AIO stats, I think it was shift + a, not sure though, easy to try it out anyway.

You could post the output of

Code:

iostat -A 2 10
# and
vmstat -wt 2 10
# and
lsattr -El aio0

zxmaus · #3 02-08-2012

and post the filesystem_io_options of oracle + oracle version + something about your disk layout - so are your filesystems setup with min or max distribution, blocksize ...
output of mount command will help and definitely mounting your oracle filesystems with noatime option and if you have a dedicated dump device with rbrw
If you dont want to use SETALL in filesystem_io_options than you might want to consider the filesystems containing oracle data + redologs to be mounted with cio, how many volumegroups with how many disks do you have and similar things
In many cases a hot disk is easily avoidable by changing your filesystems from minimum to maximum distribution and reorganize the volumegroup
I would be in addition interested in vmstat -v and vmstat -s outputs on top of what zaxxon asked for already.
Please gather all data during the time where the system is busy and slow - not during an idle timeframe or the data wont help

guzzelle · #4 02-09-2012

Thanks Zaxxon & zxmaus,
I don't know where to begin before this thread opened.

For iostat -A 2 10, vmstat -wt 2 10, vmstat -v and vmstat -s , I will post a snapshot for these once the issue occurs again.

For lsattr -El aio0 , i did't get anything so i tried lsattr -El sys0 (i hope it will do).
--> See attachment - lsattr sys0.jpg

For "Do you use asynchronous I/O (AIO) and have it tuned?"
--> I have no idea for this since I am new here and I came here in the middle of the application roll-out to production. I wish I had a clue. No knowledge on the history of the servers here.
However i checked the I/O stat in nmon and here it is:
-->

Code:

Total AIO processes=  72 Actually in use=   0  CPU used=   1.1%
         All time peak=  90     Recent peak=   7      Peak=   3.4%

guzzelle · #5 02-09-2012

if the attachment is not viewable here's the lsattr -El sys0:

Code:

SW_dist_intr    false              Enable SW distribution of interrupts              True
autorestart     true               Automatically REBOOT OS after a crash             True
boottype        disk               N/A                                               False
capacity_inc    0.01               Processor capacity increment                      False
capped          true               Partition is capped                               False
conslogin       enable             System Console Login                              False
cpuguard        enable             CPU Guard                                         True
dedicated       false              Partition is dedicated                            False
enhanced_RBAC   true               Enhanced RBAC Mode                                True
ent_capacity    6.00               Entitled processor capacity                       False
frequency       6400000000         System Bus Frequency                              False
fullcore        false              Enable full CORE dump                             True
fwversion       IBM,AL720_082      Firmware version and revision levels              False
ghostdev        0                  Recreate devices in ODM on system change          True
id_to_partition 0X80000B9662900002 Partition ID                                      False
id_to_system    0X80000B9662900000 System ID                                         False
iostat          false              Continuously maintain DISK I/O history            True
keylock         normal             State of system keylock at boot time              False
log_pg_dealloc  true               Log predictive memory page deallocation events    True
max_capacity    12.00              Maximum potential processor capacity              False
max_logname     9                  Maximum login name length at boot time            True
maxbuf          20                 Maximum number of pages in block I/O BUFFER CACHE True
maxmbuf         0                  Maximum Kbytes of real memory allowed for MBUFS   True
maxpout         8193               HIGH water mark for pending write I/Os per file   True
maxuproc        2048               Maximum number of PROCESSES allowed per user      True
min_capacity    3.00               Minimum potential processor capacity              False
minpout         4096               LOW water mark for pending write I/Os per file    True
modelname       IBM,8205-E6B       Machine name                                      False
ncargs          256                ARG/ENV list size in 4K byte blocks               True
nfs4_acl_compat secure             NFS4 ACL Compatibility Mode                       True
pre430core      false              Use pre-430 style CORE dump                       True
pre520tune      disable            Pre-520 tuning compatibility mode                 True
realmem         44826624           Amount of usable physical memory in Kbytes        False
rtasversion     1                  Open Firmware RTAS version                        False
sed_config      select             Stack Execution Disable (SED) Mode                True
systemid        IBM,020678F8P      Hardware system identifier                        False
variable_weight 0                  Variable processor capacity weight                False

zxmaus · #6 02-09-2012

From the data you provided so far, you have 1 raidset raid 10 from SAS (so internal storage) disks of a total of 1 TB (presented to the system as 1 disk) for 6 DBs and anything else running on the system excluding root - this just asks for problems as you access all your storage just with one serial path.

Even worse all your filesystems are sharing the same logfile and if I assume correctly and your filesystems are not mounted with noatime option that means that every single read (which includes as simple things as ls) and every single write of 8 different filesystems concur about access to the logfile which by nature makes this logfile naturally the hotspot of the entire system.

Still waiting for the vmstat outputs but I bet that your system has only the default filesystem tuning and is running out of buffers most of the time.

Can you post lvmo -a -v oravg output please to confirm?

Regarding aio - dont worry - on AIX 6.1 you find it with the ioo -a | grep aio command but AIX will turn it on automatically if oracle or any other application wants to use it.

filesystem_io_options is a variable set within oracle (ask your DBA) and can be set to none (standard I think in your oracle version), async or setall - the setall option lets decide oracle to use cio with async IO but wont let you access open database files outside of the database itself other than with rman which might be a problem if you dont do rman backups.

Please run a simple mount on the box to allow us to see if you are using any mount options on the filesystems.

So far

- consider to give each of your oravg filesystems its very own logfile
- consider another storage solution and a different filesystem layout if possible since 6 DBs in the same filesystem - even if this filesystem has its own logfile, are still not such a great idea. If that is not possible, than your disk will naturally stay busy since you only have one.

Regards
zxmaus

guzzelle · #7 02-10-2012

for

Code:

lvmo -a -v oravg
 
vgname = oravg
pv_pbuf_count = 512
total_vg_pbufs = 512
max_vg_pbufs = 16384
pervg_blocked_io_count = 2848
pv_min_pbuf = 512
max_vg_pbuf_count = 0
global_blocked_io_count = 2848
 
for ioo -a | grep aio
 
 
aio_active = 1
aio_maxreqs = 65536
aio_maxservers = 30
aio_minservers = 3
aio_server_inactivity = 300
posix_aio_active = 0
posix_aio_maxreqs = 65536
posix_aio_maxservers = 30
posix_aio_minservers = 3
posix_aio_server_inactivity = 300

Thread Tools	Search this Thread
Show Printable Version Email this Page Subscribe to this Thread	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Poor Disk performance on ZFS	golemico	Solaris	1	11-08-2011 08:17 AM
HW Raid poor io performance	roli8200	UNIX for Advanced & Expert Users	1	10-10-2010 04:19 AM
poor performance processing file with awk	scooter53080	UNIX for Dummies Questions & Answers	2	04-21-2009 10:29 PM
Samba on E3500 Poor Performance!!!	alex blanco	UNIX for Advanced & Expert Users	2	08-31-2002 07:16 PM