Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Should I be worried about my AIX Cluster?
Operating Systems AIX Should I be worried about my AIX Cluster? Post 302431286 by troym72 on Monday 21st of June 2010 10:18:52 AM
Old 06-21-2010
Should I be worried about my AIX Cluster?
I have a 2-node Power 7 - 16cpu - 32gb RAM - AIX 6L cluster. The production node has 10 physical cpus (40 cores) and 24gb of RAM.

The cluster has been live for 6 weeks and I'm seeing some things on the production node that I think could be of concern but wanted to get some opinions.

The application running on the server is an interface engine that is very IO Write intensive. The application does not have built-in IO caching, so we are allowing UNIX to do the disk caching.

First VMSTAT, TOPAS, NMON are showing that the OS is using far more CPU than the user processes - 5 times more. Also, the harmad process has been growing in size. It has doubled from 100mb in size to over 250mb in size in less than a month and is the largest process on the server in terms of memory usage. (see output below)

Any help/suggestions/opinions are appreciated!
Troy Morton

Code:
/>vmstat -w 5
System configuration: lcpu=40 mem=24576MB
 kthr          memory                         page                       faults           cpu
------- --------------------- ------------------------------------ ------------------ -----------
  r   b        avm        fre    re    pi    po    fr     sr    cy    in     sy    cs us sy id wa
  2   0    5615577      42426     0     0     0     0      0     0   123   9279  2041  1  3 96  0
  4   1    5615506      42335     0     0     0     0      0     0   801  55911 10328  5 22 73  0
  4   0    5616108      41412     0     0     0     0      0     0   860 495062 10209  8 22 71  0
  9   0    5615249      43162     0     0     0   232    245     0   846 691162 11318 12 25 62  0
  7   1    5615009      43278     0     0     0     0      0     0  1308 375797  9496  9 25 65  0
  2   0    5614608      43596     0     0     0     0      0     0   427  41624  4257  3 16 81  0
  1   0    5615120      43028     0     0     0     0      0     0   203  26280  2375  1  4 94  0
  4   0    5614758      43282     0     0     0     0      0     0   715  61502  8273  4 22 74  0
  6   0    5614612      42965     0     0     0     0      0     0   885  74516 10536  5 17 78  0
  4   0    5614369      42896     0     0     0     0      0     0   969  74857 11220  5 20 75  0
  4   1    5614958      42080     0     0     0     0      0     0   835  65286  8787  5 21 75  0
  4   0    5614075      42821     0     0     0     0      0     0   652  53080  6136  4 21 75  0
  2   0    5614713      42199     0     0     0     0      0     0   339  43894  3123  2 11 86  0
  2   0    5614194      42449     0     0     0     0      0     0   256  50608  2959  1  7 92  0
  2   0    5614114      42475     0     0     0     0      0     0   299  35561  4056  2  7 92  0
  5   1    5614722      40650     0   209     0    25     34     0   928  82948 11189  5 16 78  1
  4   1    5607644      46710     0   101     0    25     34     0  1503 236593 10017  6 21 72  1
  6   1    5610682      43458     0     0     0     0      0     0   781 718731  8806 11 27 62  0
  8   1    5613516      57864     0     0     0  3500   7186     0   812 616246 14411 18 28 54  0
  4   0    5613176      58085     0     0     0     0      0     0   598 130611  5711  4 19 76  0

Code:
/>ps -ef -o vsz,pid,args | sort -n | tail -10
149108 1077598 /qdx/qdx5.7/integrator/bin/hciengine -S shec -p B69Tamfion -s TOamfion
149196 1888344 /qdx/qdx5.7/integrator/bin/hciengine -S shec -p B4Tsftmd -s TOsftmd
154764 1806552 /qdx/qdx5.7/integrator/bin/hciengine -S seb -p D11Tmisysr -s TRmisysr
158828  676132 /qdx/qdx5.7/integrator/bin/hciengine -S seb -p D2Tpyxis -s TRpyxis
169448 2699290 /qdx/qdx5.7/integrator/bin/hciengine -S pem -p P1Tpemnic
172196 3469552 [hciengin]
173716  905540 /qdx/qdx5.7/integrator/bin/hciengine -S shec -p B6Tpyxis -s TOpyxis
174924  454870 /qdx/qdx5.7/integrator/bin/hciengine -S lab -p Q6Fmisysl -s FImisyslBG
245940  184622 harmad -t HACMP -n qdx_cl

Code:
/>iostat -a 5 |grep fcs
Adapter:                   Kbps      tps    Kb_read   Kb_wrtn
fcs0                     205.7      51.4          1      1028
fcs2                     244.2      62.0          0      1222
fcs0                     595.9     143.1          1      2981
fcs2                     649.6     153.7          2      3249
fcs0                     329.1      78.4          2      1644
fcs2                     357.7      84.6          0      1789
fcs0                      87.6      23.0          1       437
fcs2                     101.0      25.6          1       504
fcs2                     238.8      60.6          1      1193
fcs0                     912.8     214.3          1      4570
fcs2                     868.3     198.5          4      4344
fcs0                     1647.6     380.8          2      8237
fcs2                     1627.4     376.6          1      8137
fcs0                     720.8     160.4          3      3601
fcs2                     743.2     164.0          0      3716
fcs0                     263.4      66.2          0      1320
fcs2                     251.4      63.4          2      1258
fcs0                     159.0      41.0          1       794
fcs2                     172.2      43.2          1       860

 

10 More Discussions You Might Find Interesting

1. High Performance Computing

Building a Solaris Cluster Express cluster in a VirtualBox on OpenSolaris

Provides a description of how to set up a Solaris Cluster Express cluster in a VirtualBox on OpenSolaris. More... (0 Replies)
Discussion started by: Linux Bot
0 Replies

2. High Performance Computing

SUN Cluster Vs Veritas Cluster

Dear All, Can anyone explain about Pros and Cons of SUN and Veritas Cluster ? Any comparison chart is highly appreciated. Regards, RAA (4 Replies)
Discussion started by: RAA
4 Replies

3. AIX

Configuring new disks on AIX cluster

We run two p5 nodes running AIX 5L in a cluster mode (HACMP), both the nodes share external disk arrays. Only the primary node can access the shared disks at a given point of time. We are in the process of adding two new disks to the disk arrays so as to make them available to the existing... (3 Replies)
Discussion started by: dnicky
3 Replies

4. AIX

Breaking AIX cluster

Hello All, I was just wondering: How do I break a server cluster in an AIX 5.2 environment? Thanks. (1 Reply)
Discussion started by: bbbngowc
1 Replies

5. AIX

Aix hacmp cluster question (oracle & sap)

Hello, I was wondering if I have 3 nodes (A, B, C) all configured to startup with HACMP, but I would like to configure HACMP in such a way: 1) Node B should startup first. After the cluster successfully starts up and mounts all the filesystems, then 2) Node A, and Node C should startup ! ... (4 Replies)
Discussion started by: filosophizer
4 Replies

6. Linux

i want to install linux on my computer, but worried about viruses

hi guys, I am new to linux. I want to install it on my home computer. I have a few questions. 1) if an exploit is found on linux, how long is it before it gets patched up? My worry is that because there are not many linux users, if a big is found, then it will be a long time before others... (4 Replies)
Discussion started by: JamesByars
4 Replies

7. Solaris

Sun cluster and Veritas cluster question.

Yesterday my customer told me to expect a vcs upgrade to happen in the future. He also plans to stop using HDS and move to EMC. Am thinking how to migrate to sun cluster setup instead. My plan as follows leave the existing vcs intact as a fallback plan. Then install and build suncluster on... (5 Replies)
Discussion started by: sparcguy
5 Replies

8. Solaris

Sun cluster 4.0 - zone cluster failover doubt

Hello experts - I am planning to install a Sun cluster 4.0 zone cluster fail-over. few basic doubts. (1) Where should i install the cluster s/w binaries ?. ( global zone or the container zone where i am planning to install the zone fail-over) (2) Or should i perform the installation on... (0 Replies)
Discussion started by: NVA
0 Replies

9. AIX

[Howto] Update AIX in HACMP cluster-nodes

As i have updated a lot of HACMP-nodes lately the question arises how to do it with minimal downtime. Of course it is easily possible to have a downtime and do the version update during this. In the best of worlds you always get the downtime you need - unfortunately we have yet to find this best of... (4 Replies)
Discussion started by: bakunin
4 Replies

10. AIX

AIX Cluster Show shared file systems.

Hello, I am working on applications on an AIX 6.1 two-node cluster, with an active and passive node. Is there a command that will show me which mount points / file systems are shared and 'swing' from one node to the other when the active node changes, and which mount points are truly local to... (6 Replies)
Discussion started by: Clovis_Sangrail
6 Replies
All times are GMT -4. The time now is 06:00 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy