your IO subsystem seems to be causing your issues. Did you setup your logical volumes (filesystems, raws) with max or minimum distribution? How big are your disks? What is the output of vmstat -v and vmstat -s. What is the queue depth on your disks and if this is vio storage, on the vio servers.
What do you mean by logical volumes (filesystems, raws) with max or minimum distribution? Disks are of 100Gb x 2. queue_depth is 40 and this is XIV storage.
Code:
#vmstat -v
4194304 memory pages
3986502 lruable pages
574924 free pages
1 memory pools
413979 pinned pages
95.0 maxpin percentage
3.0 minperm percentage
90.0 maxperm percentage
64.2 numperm percentage
2561112 file pages
0.0 compressed percentage
0 compressed pages
64.2 numclient percentage
90.0 maxclient percentage
2561112 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
2228 filesystem I/Os blocked with no fsbuf
8 client filesystem I/Os blocked with no fsbuf
38788 external pager filesystem I/Os blocked with no fsbuf
25.2 percentage of memory used for computational pages
Code:
#vmstat -s
2911826 total address trans. faults
857620 page ins
6318673 page outs
0 paging space page ins
0 paging space page outs
0 total reclaims
1720594 zero filled pages faults
35814 executable filled pages faults
0 pages examined by clock
0 revolutions of the clock hand
0 pages freed by the clock
248183 backtracks
0 free frame waits
0 extend XPT waits
110954 pending I/O waits
7175994 start I/Os
2749907 iodones
22266453 cpu context switches
2734773 device interrupts
289681 software interrupts
2108993 decrementer interrupts
371 mpc-sent interrupts
371 mpc-receive interrupts
35496 phantom interrupts
0 traps
92113814 syscalls
I mean the interpolicy which you usually define during creation of your logical volumes / rawdevices. If its set to minimum - and you have only a few huge disks - you speak to your data in a serial fashion - which is usually a quite bad idea for databases with the only exception of sybase IQ and oracle asm which handle the data distribution internally.
You still did not answer if you run sybase in filesystems or rawdevices. Generally - and specifically tempdb as this is the most used part of your sybase DB. From the amount of numperm you are using, you are utilizing most of your memory for filecaching so I would guess it's filesystems - or you are having otherwise plenty of non-raw IO which is buffered though it probably doesnt have to be. You might want to consider moving your tempdb's into RAM disk after having remediated the reason for hogging so much non-comp memory.
You might want to consider as well to set j2_dynamicBufferPreallocation=128 or 256. And ... what is your network size setting in your sybase version / which sybase version do you actually run. And is it the same between p6 and p7 - or did you maybe upgrade from 12 to 15 - in which case any stored procedures you might have can cause your issues.
Few more questions:
Disks are assigned directly (FC adapters) or through VIO?
Those are 2 disks in mirror?
What is the pp and block size on you VG/filesystems.
But I agree with zxmous it does not looks like memory or cpu issue then only thing what left is storage.
Maybe you just got unlucky and got disks from pool shared with other heavy used systems.
Have you been trying to compare db IO stats a specially storage IO response time ?
The interpolicy is minimum. The sybase db and tempdb are on filesystems.
The sybase version is 12.5.2 on both live and new environment.
On P7 disk are from XIV directly attached to server and not from VIO. On old P6 the disk are from DS8k via SVC.
To zaxxon
The P7 lpars has dedicated processors same as P6, the only difference is P6 is 4Ghz and P7 s 3.5Ghz. The EOD process is absolutely same. As per the dba the database is indexed. The old environment is tuned and is on 5.3. As per the IBM documentation aix 6.1 is already tuned.
Yesterday I have disabled the multi threading and the time taken is reduced to 37 min. But its much more than live which is 25 min.
As zxmaus has suggested need to check by creating raw disk for temp db. I will check and revert.
---------- Post updated at 12:40 PM ---------- Previous update was at 12:14 PM ----------
The below are the sybase settings.
Code:
[Named Cache:abwslive_data_cache]
cache size = 750M
cache status = mixed cache
cache replacement policy = DEFAULT
local cache partition number = DEFAULT
[16K I/O Buffer Pool]
pool size = 100.0000M
wash size = DEFAULT
local async prefetch limit = DEFAULT
[4K I/O Buffer Pool]
pool size = 360.0000M
wash size = DEFAULT
local async prefetch limit = DEFAULT
[Named Cache:default data cache]
cache size = 800M
cache status = default data cache
cache replacement policy = DEFAULT
local cache partition number = DEFAULT
[16K I/O Buffer Pool]
pool size = 100.0000M
wash size = DEFAULT
local async prefetch limit = DEFAULT
[Meta-Data Caches]
number of open databases = DEFAULT
number of open objects = 4000
open object spinlock ratio = DEFAULT
number of open indexes = 700
open index hash spinlock ratio = DEFAULT
open index spinlock ratio = DEFAULT
partition groups = DEFAULT
partition spinlock ratio = DEFAULT
[Disk I/O]
disk i/o structures = 600
number of large i/o buffers = DEFAULT
page utilization percent = DEFAULT
number of devices = 30
disable disk mirroring = DEFAULT
allow sql server async i/o = DEFAULT
[SQL Server Administration]
procedure cache size = 107520
runnable process search count = 100
number of aux scan descriptors = 1000
[User Environment]
number of user connections = 500
stack size = DEFAULT
stack guard size = DEFAULT
permission cache entries = 40
user log cache size = 2560
IN solaris, for network high-availability we are using IPMP concept, can u tell me in REDHAT LINUX what we are using... also pls share good step to read & understand the that concept...
Also performance issue in linux what are step & cmd can u tell me??? (2 Replies)
Hi
We have an AIX5.3 server with application which is written in C. We are facing server (lpar) hangs intermediately. If we open new telnet window prompts for user and takes hell of a time to authenticate, not only that if we run ps -aef then also it takes lot of time. surprisingly there is no... (2 Replies)
hi I am having a performance issue with the following requirement
i have to create a permutation and combination on a set of three files
such that each record in each file is picked and the output is redirected in
a specific format but it is taking around 70 odd hours to prepare a
combination... (7 Replies)
Hi All,
I have the following script which I use in Nagios to check the health of the applications, the problem with it is that the curl part ($TOTAL) does not return anything after running for 2-3 hrs, even though from command line the script runs fine but not from Nagios.
There are 17... (1 Reply)
Hi Gurus,
I am beginner in solaris and want to know what are the things we need to check for performance monitoring on our solairs OS.
for DISK,CPU and MEMORY.
Also how we do ipforwarding in slaris
Many thanks for your help
Pradeep P (4 Replies)
In my C program i am using very large file(approx 400MB) to read parts of it frequently. But due to large file the performance of the program goes down very badly. It shows very high I/O usage and I/O wait time.
My question is, What are the ways to optimize or tune I/O on linux or how i can get... (10 Replies)
Hi,
on a linux server I have the following :
vmstat 2 10
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 4 0 675236 39836 206060 1617660 3 3 3 6 8 7 1 1 ... (1 Reply)
We have a AIX v5.3 on a p5 system with a poor performing Ingres database.
We added one CPU to the system to see if this would help. Now there are two CPU's.
with sar and topas -P I see good results: CPU usage around 30%
with topas I only see good results in the process output screen, the... (1 Reply)
Hello all,
I just stuck up in an uncertain situation related to network performance...
I am trying to access one of my remote client unix machine from a distant location..
The client machine is Ultra-5_10 , with SunOS 5.5.1
The ndd result ( hme1 )shows that the machine is hooked to a... (5 Replies)
