AIX

Hi Guys,

I would like to hear some comments about enabling Interpolicy Maximum at the LV level. I heard that when using PP striping on AIX the I/O performance get a big boost. Also the filesystem on top of these LV's/VG are hosting Databases (oracle). Your comments and any advise that you can provide will be highly appreciated . Best Regards,

Harby.

Note: Just in case we are on AIX 7.1.

Difficult to reply like that, it depends a lot on your architecture... Are you talking of external disks ? LUNS perhaps? how do you acces them (vai HBA's , how many?...)

striping can boost of course but then you are stuck with your policy... e.g. striping using 6 LUNs via 2 HBA will give great perfs, but the day you are short of space you are doomed if you cannot give 6 extra LUNS...
What follows can be achieved by letting the system use all you give:

Code:

n12:/home/vbe $ lslv -l lv111
lv111:/opt/oracle
PV                COPIES        IN BAND       DISTRIBUTION  
hdisk2            003:000:000   0%            000:000:000:003:000 
hdisk8            003:000:000   0%            000:000:000:003:000 
hdisk9            003:000:000   0%            000:000:000:003:000 
hdisk1            003:000:000   0%            000:000:000:003:000 
hdisk5            002:000:000   0%            000:000:000:000:002 
hdisk6            002:000:000   0%            000:000:000:000:002 

LOGICAL VOLUME:     lv111                  VOLUME GROUP:   vg01
LV IDENTIFIER:      00c8e5f200004c0000000123c1b0003f.13 PERMISSION:     read/write
VG STATE:           active/complete        LV STATE:       opened/syncd
TYPE:               jfs2                   WRITE VERIFY:   off
MAX LPs:            516                    PP SIZE:        128 megabyte(s)
COPIES:             1                      SCHED POLICY:   parallel
LPs:                16                     PPs:            16
STALE PPs:          0                      BB POLICY:      relocatable
INTER-POLICY:       maximum                RELOCATABLE:    yes
INTRA-POLICY:       middle                 UPPER BOUND:    6
MOUNT POINT:        /opt/oracle            LABEL:          /opt/oracle
DEVICE UID:         0                      DEVICE GID:     0
DEVICE PERMISSIONS: 432
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes (superstrict)
Serialize IO ?:     NO
INFINITE RETRY:     no
DEVICESUBTYPE:      DS_LVZ
COPY 1 MIRROR POOL: None
COPY 2 MIRROR POOL: None
COPY 3 MIRROR POOL: None

But this is not what is commonly called stripes and so the day you are short in the VG you can add what you have and let AIX deal with it...

That said, tuning a system is knowing what is the best compromise for the appointed tasks to need to realize :
Stripping is very good but it has a cost... I remember seeing an HP-UX system striped over 180 disks using 16 HBAs...
But as I said tuning is finding an acceptable compromise, e.g. bigger the cache, less memory you have for your programs ...
Then of course e.g. oracle greater the SGA better the perfs. should be, only I noticed that this is not quite true, it will follow more something like a Gauss curve and so the best can be the ennemy of the Good.... (e.g. You can end with a system loooooosing such a time parsing the SGA for some data when reading from disk would already give the data...) Also having to many Caches will end with same sort of result ( If you are using a disk bay, it caches also so why let AIX use a lot of cache for your file systems... )
About disks/ LUNs the spindles you can give and spread over the more HBAs - bigger the throughput you have, but keep in mind you may be moving the problem forward: for it to be the most performant the ideal would be to have a LUNs from separate arrays using different controllers in the bay...
All this to say take your time defining correctly what your needs are because once your system is up, you will have to live with it

Some clarifications first:

How does "striping" work?
A disk (that is: one from the last 20 years) has a small (some MBs) buffer memory. Whenever a cylinder is read the data of this cylinder are copied to this bufer and this data is therefore available relatively fast, but the disk needs some "downtime" after this to read and provide the next cylinder. If you have several disks and allocate space in a round-robin way that means that one disk after the other is queried and while it spends the time providing the next cylinder the other disks are queried. The system as a whole looks faster from the outside than any single disk could be.

What means "inter-policy maximum"?
It means that a maximum possible number of PVs (basically hdisk devices belonging to one VG) are used to allocate the PPs for a LV. If you have 5 PVs (disks) in a VG and allocate 5 PPs to a LV "maximum" would distribute them over all these PVs while "minimum" would allocate them sequentially on one disk (always given that each of these disks has enough free PPs).

How does this relate to "PP striping"?
In fact: mostly not at all nowadays. The first reason is that "disks" (what AIX knows as hdisk devices) are rarely physical disks any more but typically LUNs from a storage system or at least RAID sets with a (caching) controller. Such storage systems allocate the space they present to the outside already in such a way like described above. You can't gain any speed from striping twice but - because of a possible Moiré pattern - there is a small chance of a worsening effect (all the disk I/O effectively landing on the same physical disk).

The second reason is: many years ago there typically were PP sizes of 4 or 8MB. This is a size which fits well into the buffer memory of the hdisk. Nowadays we have way bigger PP sizes (see the example above, 128MB in vbe's posting is rather at the lower end) and these big PPs won't fit into drives buffer memory, therefore you won't gain only a little speed because the disk, after having provided the "fast data", will have to provide some "slow data" before I/O changes to another disk.

There is even a third reason: modern controllers have awful lots of cache, typically 1-several GB. These big caches make striping obsolete, because any effect striping could have would not be noticeable anyway.

A final aspect: to benefit from striping the most the access pattern should be sequential because this makes sure the access happens evenly distributed across all disks. You mention that you use a RDBMS (Oracle), so the overwhelming majority of disk access will be not sequential at all but random access. To speed such a random access striping will not help you (or very little at all), the most you will get out of some cache. If you prefer cache to be in hardware (caching controller, etc.) or software (SGA, Unix buffering I/O via "file memory", etc.) ia rather a matter of taste. I guess there won't be all too much difference between equal amounts of cache memory cming from any of these sources.

I hope this helps.

bakunin

Thanks Guys I really appreciate your valuable comments. The reason behind my posting was because we had a while ago an IBM expert doing some work with the Midrange/Storage team and according to him and what he has seen across different customer sites is that by implementing PP striping has made a big difference in performance. As you guys mentioned in your posts nowdays we have plenty of cache almost at every single level (lpar,storage subsystem) and stripping might not help as it used to do it in the past.

In our case when it comes to achieve better performance for databases we try to avoid double buffering by bypassing the filesystem cache hence making the filesystem cache smaller and that extra memory can be allocated by the Oracle SGA. On AIX we rely heavily on Concurrent I/O. Once Concurrent I/O is on Direct reads/writes are mostly used by the database and locking issues caused by inode locking on a heavily updated systems are a thing of the past. Having said this we have seen performance improvements by allocating different LUN's sizes (33/100/500 Gigs) depending on the system (OLTP,DSS) and adjusting QDEPTH. We have seen single LUN achieve more than 3000 IOPS. We'll be performing Performance and Volume test and it would be insteresting to see the results using different types of secenarios.

Thanks Guys....

Thank you for your follow-up and explanation. We enjoy getting feedback like this and enhance our own knowledge by getting to know what worked, what didn't and why.

...which in turn uses this extra memory for cache, LOL. Still, i suppose the Oracle database knows better than the OS how its own data can be cached most effectively, so taking from the OS and giving to the database makes sense.

I for my part (and most probably all of my colleagues here) would be very interested in these performance tests. PLEASE post them, we'd be eternally grateful and perhaps eager to discuss them in-depth. Whatever you may need for this, just ask me or zaxxon (we are the AIX people in the team, any other staff member will help you too, of course) and we will be glad to help.

bakunin

Of course I'll get back after we complete the test. It might take me a few weeks before we get there but I will post the results. The discussions in this forum are very very helpul. Thanks again for your feedback and help.