Login or Register to Ask a Question and Join Our Community


AIX

Performance issue

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems AIX Performance issue
# 1  
Old 12-04-2012
Reference Performance issue
Hi

We have an AIX5.3 server with application which is written in C. We are facing server (lpar) hangs intermediately. If we open new telnet window prompts for user and takes hell of a time to authenticate, not only that if we run ps -aef then also it takes lot of time. surprisingly there is no cpu bottleneck as cpu is almost 80% idle, there is o memory or paging, there are no errors on network card. The only thing we observed is there were so many errors of core dump for the application.
Code:
                           Kbps      tps    Kb_read   Kb_wrtn  time
                         585.2      14.8        288     17264  09:57:51
Adapter:                      xfers              time
-------------------- --------------------------- ---------
                        bps    tps  bread  bwrtn
sissas0              598.9K  14.7    9.8K 589.1K  09:57:51

Disks:                           xfers                                read                                write                                  queue                    time
-------------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------
                       %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv
                       act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull
hdisk0                 3.1 302.1K   7.9   7.5K 294.6K   1.5   4.5    2.6    7.9     0    0   6.5   5.4    1.9    7.9     0    0   0.0    0.0    0.0    0.0   0.0   0.0  09:57:51
hdisk1                 2.8 296.9K   6.8   2.3K 294.6K   0.4   6.8    3.4   11.8     0    0   6.5   5.5    2.5   11.3     0    0   0.0    0.0    0.0    0.0   0.0   0.0  09:57:51

HTML Code:
                           Kbps      tps    Kb_read   Kb_wrtn  time
                         369.8      68.2         64     11030  09:58:21
Adapter:                      xfers              time
-------------------- --------------------------- ---------
                        bps    tps  bread  bwrtn
sissas0              378.7K  68.2    2.2K 376.5K  09:58:21

Disks:                           xfers                                read                                write                                  queue                    time
-------------------- -------------------------------- ------------------------------------ ------------------------------------ -------------------------------------- ---------
                       %tm    bps   tps  bread  bwrtn   rps    avg    min    max time fail   wps    avg    min    max time fail    avg    min    max   avg   avg  serv
                       act                                    serv   serv   serv outs              serv   serv   serv outs        time   time   time  wqsz  sqsz qfull
hdisk0                14.9 190.9K  34.4   2.0K 188.9K   0.4   4.2    0.0    0.0     0    0  34.0  19.2    1.5    8.5     0    0  22.8    0.0    0.0    0.5   0.3  10.6  09:58:21
hdisk1                14.5 187.8K  33.8 136.5  187.6K   0.0  17.1    0.0    0.0     0    0  33.8  18.4    1.3    4.6     0    0  20.8    0.0    0.0    0.5   0.3  10.2  09:58:21
"vmstat output"

Code:
System configuration: lcpu=2 mem=5376MB

   kthr            memory                         page                       faults           cpu       time  
----------- --------------------- ------------------------------------ ------------------ ----------- --------
  r   b   p        avm        fre    fi    fo    pi    po    fr     sr    in     sy    cs us sy id wa hr mi se
  3   0   0    1213172      14247    30     0     0     0     0      0  3710 115624 11911 53 31 13  3 09:57:13
  2   0   0    1213960      13119    65   256     0     0     0      0  1819 223286  7149 34 59  5  3 09:57:14
  2   0   0    1224245       2788    46   704     0     0   723   1716  3080 938143 11432 52 43  4  0 09:57:15
  4   0   0    1227928       1085    16    32     0     0  2081   4502  2285  96142 15999 28 66  5  0 09:57:16
  2   1   0    1236307       1531     0    13     0     0  9000  24997  1031  14441  3713  9 37 53  1 09:57:17
  0   0   0    1224268      14239    12     1     0     0   508   1319  3107  56542 12801 24 29 46  1 09:57:18
  2   0   0    1236983       2272     0     0     0     0   923   3437  2748  54970 10001 24 22 54  0 09:57:19
  1   0   0    1217684      21522    45   737     0     0   256   1375  3546 155013 13750 31 34 34  0 09:57:20
  4   0   0    1214541      24604     8    64     0     0     0      0  4082  86059 17777 30 44 21  5 09:57:21
  0   1   0    1211946      27162     0    36     0     0     0      0  4236  36690 14955 27 18 47  7 09:57:22
  1   0   0    1211769      27338     0     0     0     0     0      0  3716  26524 14050 22 20 58  0 09:57:23
  1   0   0    1211963      27143     0     0     0     0     0      0  3645  47245 14894 24 23 53  0 09:57:24
  0   0   0    1212044      27029    17     0     0     0     0      0  5092  42021 18800 34 26 39  0 09:57:25
  3   0   0    1212056      27026     1     0     0     0     0      0  7440  70306 29180 36 26 38  0 09:57:26
  0   0   0    1212064      27018     0     0     0     0     0      0  4734  37139 18375 28 25 47  0 09:57:27
  2   0   0    1212146      26936     0    15     0     0     0      0  4068  22927 14339 25 18 57  0 09:57:28
  0   0   0    1212149      24933     0  1984     0     0     0      0  4928  34577 17412 36 25 39  0 09:57:29
  0   0   0    1212147      24936     0     0     0     0     0      0  5531  32206 19008 40 18 42  0 09:57:30
  0   0   0    1212149      24933     0     0     0     0     0      0  3785  26694 13758 23 22 55  0 09:57:31
  0   0   0    1212141      24940     1     0     0     0     0      0  3908  21671 14132 23 15 62  0 09:57:32
   kthr            memory                         page                       faults           cpu       time  
----------- --------------------- ------------------------------------ ------------------ ----------- --------
  r   b   p        avm        fre    fi    fo    pi    po    fr     sr    in     sy    cs us sy id wa hr mi se
  1   0   0    1212152      24933     0     0     0     0     0      0  3858  37512 14007 24 22 54  0 09:57:33
  2   0   0    1212697      24379    10     1     0     0     0      0  3921  28358 13789 26 18 56  1 09:57:34
  1   0   0    1212706      24370     0     0     0     0     0      0  4285  31408 15405 35 17 49  0 09:57:35
  2   0   0    1212711      24364     1     0     0     0     0      0  3454  29837 12735 52 22 27  0 09:57:36
  0   0   0    1212772      24302     0     0     0     0     0      0  4121  27853 15377 28 16 56  0 09:57:37
  6   0   0    1212978      24095     0     1     0     0     0      0  3263  26979 14024 35 18 47  0 09:57:38
  1   0   0    1213111      23960     2     0     0     0     0      0  3877  31711 13522 24 16 60  0 09:57:39
  1   0   0    1213119      23952     0     0     0     0     0      0  3350  24652 13280 20 20 60  0 09:57:40
  1   0   0    1213191      23880     0     0     0     0     0      0  3833  21134 13018 23 15 62  0 09:57:41
  1   0   0    1213193      23874     0     0     0     0     0      0  4033  30411 14517 29 22 49  0 09:57:42
  1   0   0    1213195      23868     1     0     0     0     0      0  4117  25976 14614 29 18 52  0 09:57:43
  1   0   0    1213199      23828    55     0     0     0     0      0  3979  38583 14675 27 22 50  1 09:57:44
  0   0   0    1213316      23711     0     0     0     0     0      0  3930  90174 13890 25 18 57  0 09:57:45
  1   0   0    1213311      23713     0     0     0     0     0      0  3793  27332 13836 29 21 50  0 09:57:46
  0   0   0    1213362      23654     5     0     0     0     0      0  3891  31158 13841 28 16 56  0 09:57:47
  2   0   0    1213537      23480     0     1     0     0     0      0  3290  54229 11802 28 21 51  0 09:57:48
  0   0   0    1213579      23437     0     0     0     0     0      0  3894  30331 14546 23 16 61  0 09:57:49
  0   0   0    1213423      23589     2     1     0     0     0      0  4031  32409 15190 32 22 45  0 09:57:50
  1   0   0    1211081      25922     8     0     0     0     0      0  4285  24381 15064 29 17 54  1 09:57:51
  3   0   0    1212575      24427     0     0     0     0     0      0  4033  27862 14775 28 19 53  0 09:57:52
   kthr            memory                         page                       faults           cpu       time  
----------- --------------------- ------------------------------------ ------------------ ----------- --------
  r   b   p        avm        fre    fi    fo    pi    po    fr     sr    in     sy    cs us sy id wa hr mi se
  0   0   0    1211471      25528    24     0     0     0     0      0  4032  26727 15380 25 18 57  0 09:57:53
  0   0   0    1211471      25528     0     0     0     0     0      0  3678  20203 12995 23 15 63  0 09:57:54
  0   0   0    1211483      25514     0     0     0     0     0      0  3284  32100 14016 21 21 58  0 09:57:55
  2   0   0    1212630      24364     0     0     0     0     0      0  2705  23693 11065 18 13 69  0 09:57:56
  1   0   0    1212788      24210     0     0     0     0     0      0  4169  32903 15619 28 24 48  0 09:57:57
  0   0   0    1213009      23986    14     1     0     0     0      0  1806  12979  7245 11  8 80  0 09:57:58
  2   0   0    1213014      23981     0     0     0     0     0      0   949  12599  4039 10 13 78  0 09:57:59
  0   0   0    1213116      23878     0     0     0     0     0      0  2181  13214  7538 19  9 72  0 09:58:00
  0   0   0    1213232      23761     0     0     0     0     0      0  2380  21901  9532 19 17 63  0 09:58:01
  0   0   0    1213277      23716     0     0     0     0     0      0  1350   9198  4559 15  6 78  0 09:58:02
  0   0   0    1213284      23708     0    23     0     0     0      0  1517  14331  5235 14 15 61 10 09:58:03
  0   0   0    1213410      23577     0    60     0     0     0      0  4688  29454 16426 38 22 26 14 09:58:04
  2   1   0    1213430      23555     0   222     0     0     0      0  3169  28903 11793 24 21 39 16 09:58:05
  1   0   0    1213428      23557     0   176     0     0     0      0  3979  21043 13045 26 18 36 20 09:58:06
  1   0   0    1213429      23555     0    10     0     0     0      0  4348  29102 14229 24 25 49  3 09:58:07
  0   1   0    1213437      23547     0     1     0     0     0      0  3016  45648  9699 23 14 58  5 09:58:08
  0   0   0    1212750      24232     0     0     0     0     0      0  1601  23978  8712 20 16 64  0 09:58:09
  0   0   0    1212478      24503     0     0     0     0     0      0   628  43858  4317 15 19 67  0 09:58:10
  1   0   0    1213179      23797     1     0     0     0     0      0  2568  50097  9130 28 20 51  0 09:58:11
  0   0   0    1214633      22342     0     0     0     0     0      0  4052  23891 13778 31 17 52  0 09:58:12
   kthr            memory                         page                       faults           cpu       time  
----------- --------------------- ------------------------------------ ------------------ ----------- --------
  r   b   p        avm        fre    fi    fo    pi    po    fr     sr    in     sy    cs us sy id wa hr mi se
  0   0   0    1217873      19102     0     0     0     0     0      0  3865  31917 13392 31 18 51  0 09:58:13
  0   0   0    1217676      19289    10     0     0     0     0      0  3464  24873 12013 41 17 42  0 09:58:14
  2   0   0    1218242      18683     0    32     0     0     0      0  3843  35720 14061 29 18 53  0 09:58:15
  0   0   0    1218247      18678     0     0     0     0     0      0  3400  27049 13059 21 19 60  0 09:58:16
  1   0   0    1218229      18696     0     0     0     0     0      0  4125  26827 16139 24 16 60  0 09:58:17
  1   0   0    1218466      18458     0     1     0     0     0      0  5322  76074 20456 35 26 38  0 09:58:18
  0   0   0    1218462      18462     0     0     0     0     0      0  3161  31550 12481 21 15 65  0 09:58:19
  3   0   0    1218486      18435     0     0     0     0     0      0  5019  48378 19111 31 24 45  0 09:58:20
  2   0   0    1218278      18644     0     0     0     0     0      0  5803  68820 21453 38 22 40  0 09:58:21
  2   0   0    1218271      18650     0     0     0     0     0      0  4946  45197 18728 31 23 46  0 09:58:22
  3   0   0    1218326      18596     0     0     0     0     0      0  3750  26586 15387 24 15 61  0 09:58:23
  2   0   0    1218544      18378     0     0     0     0     0      0  2809  29751 12687 26 18 56  0 09:58:24
  0   0   0    1219093      17826     0     0     0     0     0      0  2700  25927 11668 26 14 60  0 09:58:25
  1   0   0    1219730      17185     0     0     0     0     0      0  2939  32015 13081 29 21 50  0 09:58:26
  1   0   0    1219738      17177     0     0     0     0     0      0  2765  21737 11996 23 13 64  0 09:58:27
  0   0   0    1219947      16967     0     5     0     0     0      0  2824  27244 11965 30 20 50  0 09:58:28
  1   0   0    1219943      16967     5     2     0     0     0      0  4182  37190 16750 39 18 43  0 09:58:29
  3   0   0    1220099      16811     0     0     0     0     0      0  5989  44857 22535 38 25 37  0 09:58:30
  2   0   0    1220931      15978     0     0     0     0     0      0  5692  40823 21709 44 21 35  0 09:58:31
  0   0   0    1221438      15470     8     0     0     0     0      0  5537  56461 21453 36 27 37  0 09:58:32
   kthr            memory                         page                       faults           cpu       time  
----------- --------------------- ------------------------------------ ------------------ ----------- --------
  r   b   p        avm        fre    fi    fo    pi    po    fr     sr    in     sy    cs us sy id wa hr mi se
  1   0   0    1220941      15966     1     0     0     0     0      0  5946  49800 21525 43 23 34  0 09:58:33
  2   0   0    1221772      15135     0     0     0     0     0      0  6246  58446 23431 37 23 40  0 09:58:34
  6   0   0    1221608      15304     0     0     0     0     0      0  4981  77505 19651 38 28 35  0 09:58:35
  3   0   0    1221741      15158    18     1     0     0     0      0  4802 229972 18775 36 31 33  0 09:58:36
  3   0   0    1222282      14616     0     2     0     0     0      0  4094 173709 16648 35 30 34  0 09:58:37
  2   0   0    1222632      14249    13     4     0     0     0      0  3331 140823 13253 31 26 40  2 09:58:38
  1   0   0    1223227      13652     0     0     0     0     0      0  3093  55930 12104 30 20 50  0 09:58:39
  0   0   0    1223519      13357     0     6     0     0     0      0  5113  39177 18264 35 19 45  1 09:58:40
  1   0   0    1223559      13317     0     6     0     0     0      0  6152  58235 23403 36 27 37  0 09:58:41
  0   0   0    1223652      13222     9     0     0     0     0      0  2011  19324  7381 21 10 69  0 09:58:42
  0   0   0    1223876      12999     0     0     0     0     0      0  2494  40136 10229 23 19 58  0 09:58:43
  1   0   0    1224076      12799     0     0     0     0     0      0  5877  46555 21618 35 20 45  0 09:58:44
  0   0   0    1222992      13879     0     2     0     0     0      0  5951  49562 23099 34 28 38  0 09:58:45
  0   0   0    1223136      13663    74    10     0     0     0      0  5127  32905 18769 30 18 51  0 09:58:46
  1   0   0    1222404      14397     0     0     0     0     0      0  5520  84529 21312 33 27 39  0 09:58:47
  0   0   0    1222452      14347     0     1     0     0     0      0  5349  89317 20386 29 22 48  0 09:58:48
  0   0   0    1222120      14652    26     0     0     0     0      0  3896  32365 16100 24 21 54  1 09:58:49
  1   0   0    1222142      14618    12     0     0     0     0      0  6049  48556 22919 34 21 45  0 09:58:50
  1   0   0    1222304      14454     0     0     0     0     0      0  6668  73227 26926 36 30 34  0 09:58:51
  1   0   0    1221187      15567     8     2     0     0     0      0  6228  94519 24593 36 29 34  0 09:58:52
   kthr            memory                         page                       faults           cpu       time  
----------- --------------------- ------------------------------------ ------------------ ----------- --------
  r   b   p        avm        fre    fi    fo    pi    po    fr     sr    in     sy    cs us sy id wa hr mi se
  1   0   0    1222168      14582     0     0     0     0     0      0  5800 165168 23250 36 30 34  0 09:58:53
  2   0   0    1220658      16094     0     0     0     0     0      0  4652 214673 18782 34 35 30  0 09:58:54
  0   0   0    1220683      16067     0     0     0     0     0      0  4089  49429 16681 25 18 57  0 09:58:55
  0   0   0    1220683      16067     0     0     0     0     0      0  5573  43567 22323 29 25 46  0 09:58:56
  1   0   0    1220729      16008    22     0     0     0     0      0  3380  25609 12853 24 13 62  1 09:58:57
  3   0   0    1220932      15803     0     1     0     0     0      0  3531  41507 13943 24 21 54  0 09:58:58
  4   0   0    1218686      18047     0     0     0     0     0      0  3117  22044 11402 26 14 60  0 09:58:59
  2   0   0    1215429      21302     0     3     0     0     0      0  3148  27815 12031 25 21 54  0 09:59:00
  3   0   0    1215424      21307     0     0     0     0     0      0  4444  27695 16145 30 16 54  0 09:59:01
  1   0   0    1216411      20314     0     0     0     0     0      0  3855  33125 14212 34 21 44  0 09:59:02
  1   0   0    1216850      19875     0     0     0     0     0      0  4572  51358 17211 34 18 48  0 09:59:03
  2   0   0    1217038      19684    16     0     0     0     0      0  3853  31791 14190 36 21 43  0 09:59:04
  2   0   0    1216973      19656     3    32     0     0     0      0  5064  40493 19548 42 19 39  0 09:59:05
  0   0   0    1217087      19374     0   192     0     0     0      0  5531  45886 22204 42 25 33  0 09:59:06
  1   1   0    1217172      19120     0   214     0     0     0      0  6359  47394 24709 42 24 25 10 09:59:07
  2   1   0    1217169      18952    12   249     0     0     0      0  4963  39831 19332 39 26 21 13 09:59:08
  2   1   0    1217172      18779     0   228     0     0     0      0  6391  44199 24828 41 25 24 10 09:59:09
  2   0   0    1217357      18437     0   191     0     0     0      0  6586  62367 26637 46 28 26  0 09:59:10
  1   0   0    1217362      18257     8   128     0     0     0      0  5587  59916 24099 39 27 34  0 09:59:11
  0   0   0    1217441      17991     0   224     0     0     0      0  3815  26898 16012 34 17 49  0 09:59:12

please suggest.
Last edited by Scott; 12-04-2012 at 02:11 PM.. Reason: Changed ICODE tags to CODE tags
# 2  
Old 12-05-2012
Quote:
We have an AIX5.3 server with application which is written in C. We are facing server (lpar) hangs intermediately. If we open new telnet window prompts for user and takes hell of a time to authenticate, not only that if we run ps -aef then also it takes lot of time.
Authentication usually involves a third machine - DNS server, LDAP server, etc.. It is possible that this problem lies there.

Quote:
surprisingly there is no cpu bottleneck as cpu is almost 80% idle, there is o memory or paging, there are no errors on network card. The only thing we observed is there were so many errors of core dump for the application.
Have you tried to analyze these cores? Install the fileset "bos.adt.debug" and use "dbx" to analyze the core.

I hope this helps.

bakunin
# 3  
Old 12-17-2012
We have resoled this issue by removing one AD server entry from ldap configuration. Don't know exact reason, but think if you have multiple entries in ldap config file and one of the server is having issue then it stalls the login requests.

powerAIX
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

Performance issue in Linux

IN solaris, for network high-availability we are using IPMP concept, can u tell me in REDHAT LINUX what we are using... also pls share good step to read & understand the that concept... Also performance issue in linux what are step & cmd can u tell me??? (2 Replies)
Discussion started by: tiger09
2 Replies

2. AIX

Performance issue

Hi, We have 2 lpars on p6 blade. One of the lpar is having 3 core cpu with 5gb memory running sybase as database. An EOD process takes 25 min. to complete. Now we have an lpar on P7 server with entitled cpu capacity of 2 with 16 Gb memory and sybase as database. The EOD process which takes... (17 Replies)
Discussion started by: vjm
17 Replies

3. UNIX for Dummies Questions & Answers

Performance issue

hi I am having a performance issue with the following requirement i have to create a permutation and combination on a set of three files such that each record in each file is picked and the output is redirected in a specific format but it is taking around 70 odd hours to prepare a combination... (7 Replies)
Discussion started by: mad_man12
7 Replies

4. Shell Programming and Scripting

Performance issue or something else?

Hi All, I have the following script which I use in Nagios to check the health of the applications, the problem with it is that the curl part ($TOTAL) does not return anything after running for 2-3 hrs, even though from command line the script runs fine but not from Nagios. There are 17... (1 Reply)
Discussion started by: jacki
1 Replies

5. Solaris

Performance issue

Hi Gurus, I am beginner in solaris and want to know what are the things we need to check for performance monitoring on our solairs OS. for DISK,CPU and MEMORY. Also how we do ipforwarding in slaris Many thanks for your help Pradeep P (4 Replies)
Discussion started by: ppandey21
4 Replies

6. UNIX for Advanced & Expert Users

Performance issue!

In my C program i am using very large file(approx 400MB) to read parts of it frequently. But due to large file the performance of the program goes down very badly. It shows very high I/O usage and I/O wait time. My question is, What are the ways to optimize or tune I/O on linux or how i can get... (10 Replies)
Discussion started by: mavens
10 Replies

7. Shell Programming and Scripting

performance issue

I want to read a file. is it good to use File I/O or shell script?? which one is the best option? (1 Reply)
Discussion started by: vishwaraj
1 Replies

8. UNIX for Advanced & Expert Users

performance issue

Hi, on a linux server I have the following : vmstat 2 10 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 4 0 675236 39836 206060 1617660 3 3 3 6 8 7 1 1 ... (1 Reply)
Discussion started by: big123456
1 Replies

9. AIX

performance issue

We have a AIX v5.3 on a p5 system with a poor performing Ingres database. We added one CPU to the system to see if this would help. Now there are two CPU's. with sar and topas -P I see good results: CPU usage around 30% with topas I only see good results in the process output screen, the... (1 Reply)
Discussion started by: rein
1 Replies

10. UNIX for Advanced & Expert Users

Performance issue

Hello all, I just stuck up in an uncertain situation related to network performance... I am trying to access one of my remote client unix machine from a distant location.. The client machine is Ultra-5_10 , with SunOS 5.5.1 The ndd result ( hme1 )shows that the machine is hooked to a... (5 Replies)
Discussion started by: shibz
5 Replies
Login or Register to Ask a Question