AIX

Hi,

I am seeing a strange behaviour with my discks on AIX. On a Sar output i see discs are more than 100 % busy but there are no reads and writes. And during this time my applicaiton freezes and the application response time increases from 3 sec to 100+ seconds. But when the disc recovers to normal every thing will get back to normal.

The issue is happening randomly and this time on hdisk13. Please help me understand this output.

06:21:26 hdisk0 0 0.0 0 0 0.0 0.0
hdisk5 0 0.0 0 0 0.0 0.0
hdisk6 0 0.0 0 0 0.0 0.0
hdisk7 0 0.0 0 0 0.0 0.0
hdisk1 0 0.0 0 0 0.0 0.0
hdisk8 0 0.0 0 0 0.0 0.0
hdisk2 0 0.0 0 0 0.0 0.0
hdisk9 0 0.0 2 76 0.0 0.7
hdisk3 0 0.0 0 0 0.0 0.0
hdisk10 0 0.0 0 0 0.0 0.0
hdisk4 0 0.0 0 0 0.0 0.0
hdisk21 0 0.0 0 0 0.0 0.0
hdisk11 2 0.0 4 86 0.0 0.8
hdisk12 0 0.0 2 67 0.0 0.7
hdisk14 0 0.0 0 0 0.0 0.0
hdisk13 240 0.0 0 0 0.0 0.0
hdisk15 0 0.0 0 0 0.0 0.0
hdisk17 0 0.0 2 76 0.0 0.7
hdisk16 0 0.0 0 0 0.0 0.0
hdisk19 2 0.0 16 115 0.0 0.5
hdisk18 0 0.0 2 76 0.0 0.7
hdisk20 0 0.0 0 0 0.0 0.0

06:21:27 hdisk0 0 0.0 0 0 0.0 0.0
hdisk5 0 0.0 0 0 0.0 0.0
hdisk6 0 0.0 0 0 0.0 0.0
hdisk7 0 0.0 0 0 0.0 0.0
hdisk1 0 0.0 0 0 0.0 0.0
hdisk8 0 0.0 0 0 0.0 0.0
hdisk2 0 0.0 0 0 0.0 0.0
hdisk9 0 0.0 0 0 0.0 0.0
hdisk3 0 0.0 2 19 0.0 0.5
hdisk10 0 0.0 0 0 0.0 0.0
hdisk4 0 0.0 0 0 0.0 0.0
hdisk21 0 0.0 0 0 0.0 0.0
hdisk11 2 0.0 2 19 0.0 0.6
hdisk12 0 0.0 0 0 0.0 0.0
hdisk14 0 0.0 0 0 0.0 0.0
hdisk13 240 0.0 0 0 0.0 0.0
hdisk15 0 0.0 0 0 0.0 0.0
hdisk17 0 0.0 0 0 0.0 0.0
hdisk16 0 0.0 0 0 0.0 0.0
hdisk19 0 0.0 24 134 0.0 0.4
hdisk18 0 0.0 0 0 0.0 0.0
hdisk20 2 0.0 26 124 0.0 0.4

06:21:28 hdisk0 2 0.0 122 854 0.2 0.6
hdisk5 0 0.0 0 0 0.0 0.0
hdisk6 0 0.0 2 19 0.0 0.6
hdisk7 0 0.0 7 57 0.0 0.6
hdisk1 0 0.0 0 0 0.0 0.0
hdisk8 0 0.0 0 0 0.0 0.0
hdisk2 0 0.0 0 0 0.0 0.0
hdisk9 0 0.0 2 19 0.0 0.4
hdisk3 0 0.0 0 0 0.0 0.0
hdisk10 2 0.0 7 57 0.0 0.6
hdisk4 0 0.0 0 0 0.0 0.0
hdisk21 0 0.0 0 0 0.0 0.0
hdisk11 0 0.0 4 38 0.0 0.5
hdisk12 0 0.0 0 0 0.0 0.0
hdisk14 0 0.0 2 9 0.0 0.5
hdisk13 240 0.0 4 19 0.0 0.4
hdisk15 0 0.0 0 0 0.0 0.0
hdisk17 0 0.0 0 0 0.0 0.0
hdisk16 0 0.0 0 0 0.0 0.0
hdisk19 0 0.0 0 0 0.0 0.0
hdisk18 0 0.0 0 0 0.0 0.0
hdisk20 0 0.0 84 614 0.0 0.4

06:21:29 hdisk0 3 0.0 14 62 0.6 3.2
hdisk5 0 0.0 2 16 0.0 1.8
hdisk6 3 0.0 5 32 0.0 7.5
hdisk7 3 0.0 6 30 0.0 6.8
hdisk1 0 0.0 5 32 0.3 1.6
hdisk8 3 0.0 8 55 0.0 4.2
hdisk2 0 0.0 3 16 0.0 1.3
hdisk9 2 0.0 3 14 0.0 6.5
hdisk3 0 0.0 2 13 0.0 0.9
hdisk10 0 0.0 0 1 0.0 0.4
hdisk4 2 0.0 5 26 0.0 4.6
hdisk21 0 0.0 0 0 0.0 0.0
hdisk11 0 0.0 0 0 0.0 0.0
hdisk12 0 0.0 0 7 0.0 1.0
hdisk14 0 0.0 1 22 0.0 1.1
hdisk13 8 0.0 5 60 0.0 1139.1
hdisk15 0 0.0 0 4 0.0 0.8
hdisk17 0 0.0 2 14 0.0 0.6
hdisk16 0 0.0 0 1 0.0 1.1
hdisk19 2 0.0 4 16 0.0 5.8
hdisk18 0 0.0 0 0 0.0 0.0
hdisk20 2 0.0 6 32 0.0 3.7

06:21:30 hdisk0 0 0.0 0 0 0.0 0.0
hdisk5 21 0.0 126 586 0.0 1.7
hdisk6 10 0.0 44 264 0.3 3.3
hdisk7 13 0.0 66 370 0.2 2.6
hdisk1 0 0.0 0 0 0.0 0.0
hdisk8 22 0.0 137 718 0.3 2.3
hdisk2 0 0.0 5 21 0.0 0.6
hdisk9 6 0.0 44 311 0.0 1.4
hdisk3 21 0.0 466 5746 0.0 0.6
hdisk10 0 0.0 0 0 0.0 0.0
hdisk4 14 0.0 255 1411 0.0 0.6
hdisk21 0 0.0 0 0 0.0 0.0
hdisk11 2 0.0 21 100 0.0 2.8
hdisk12 5 0.0 29 253 0.0 2.1
hdisk14 6 0.0 126 2283 0.0 0.6
hdisk13 7 0.0 132 1982 0.0 0.6
hdisk15 10 0.0 215 2331 0.0 0.5
hdisk17 9 0.0 192 777 0.0 0.4
hdisk16 1 0.0 105 2199 0.0 0.6
hdisk19 0 0.0 0 0 0.0 0.0
hdisk18 0 0.0 0 0 0.0 0.0
hdisk20 5 0.0 159 639 0.0 0.5

06:21:31 hdisk0 0 0.0 0 0 0.0 0.0
hdisk5 1 0.0 2 10 0.0 0.3
hdisk6 0 0.0 0 0 0.0 0.0
hdisk7 0 0.0 28 112 0.0 0.3
hdisk1 0 0.0 0 0 0.0 0.0
hdisk8 0 0.0 0 0 0.0 0.0
hdisk2 0 0.0 1 16 0.0 0.5
hdisk9 0 0.0 0 0 0.0 0.0
hdisk3 1 0.0 82 826 0.0 0.4
hdisk10 0 0.0 0 0 0.0 0.0
hdisk4 0 0.0 81 325 0.0 0.4
hdisk21 0 0.0 0 0 0.0 0.0
hdisk11 0 0.0 18 74 0.0 0.4
hdisk12 0 0.0 0 0 0.0 0.0
hdisk14 8 0.0 170 746 0.0 0.4
hdisk13 5 0.0 156 730 0.0 0.4
hdisk15 9 0.0 225 1120 0.0 0.5
hdisk17 2 0.0 90 410 0.0 0.4
hdisk16 10 0.0 218 1098 0.0 0.4
hdisk19 16 0.0 265 1114 0.0 0.4
hdisk18 14 0.0 300 1200 0.0 0.4
hdisk20 9 0.0 213 880 0.0 0.5

Thanks,
Sreekanth

is this a local or san/iscsi disk?


what does errpt say?

Hi,

Thanks for replying. The AIX machine uses a SAN storage from EMC. I have just looked at the errpt and found a msg that is meaningfull. All other errors talked about some abnormal program termination.

LABEL: J2_FS_FULL
IDENTIFIER: F7FA22C9
Date/Time: Wed Jun 13 16:06:15 CDT 2012
Sequence Number: 83
Machine Id: 00005C6BD400
Node Id: ux9013
Class: O
Type: INFO
WPAR: Global
Resource Name: SYSJ2
Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
Probable Causes
FILE SYSTEM FULL
Recommended Actions
INCREASE THE SIZE OF THE ASSOCIATED FILE SYSTEM
REMOVE UNNECESSARY DATA FROM FILE SYSTEM
USE FUSER UTILITY TO LOCATE UNLINKED FILES STILL REFERENCED
Detail Data
JFS2 MAJOR/MINOR DEVICE NUMBER
0022 0008
FILE SYSTEM DEVICE AND MOUNT POINT
/dev/archdest_lv, /archdest

vio mapped,npiv or directly attached?

please provide all this information in your first post next time..

Hi,

Sorry for this partial info. I am not a system admin so i am not able to provide you all the info required in one shot. Also i dont know this infomation so i will get it from my admin team and reply as soon as possible.

Thanks,
Sreekanth