Login or Register to Ask a Question and Join Our Community


AIX

AIX 7.1 on Power 750 is very slow with no high memory or process consumption

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems AIX AIX 7.1 on Power 750 is very slow with no high memory or process consumption
# 1  
Old 03-04-2014
AIX 7.1 on Power 750 is very slow with no high memory or process consumption
Hi,

This thread has been posted before on linuxquestions.org, but no answer, maybe because this is unix question and not linux. I'm posting the same thread here, hope I can get an answer from someone in the meantime, I wish I could post of emergency thread but it needs bits which I don't have :

I'm posting this thread because first I'm a newbie on AIX (I've been a Linux Admin for almost 3 years and we have AIX for some project requirements) and second I could not find any useful info on that over the internet.
These servers are handed over to us by the client. We are experiencing slow performance on the console, on login (using Putty or any shell client on windows, the password pops up with a big latency) and on our Application Server (Oracle Weblogic 10.3.6). Here are the output of some of the usual commands :

Code:
-bash-4.2# svmon -G
               size       inuse        free         pin     virtual   mmode
memory     16842752     1459856    15382896     1291104     1392007     Ded
pg space    4325376        4707

               work        pers        clnt       other
pin          719376           0           0      571728
in use      1392007           0       67849

PageSize   PoolSize       inuse        pgsp         pin     virtual
s    4 KB         -      884208        4707      761888      816359
m   64 KB         -       35978           0       33076       35978

And here is the topas sample output:

Code:
Tue Mar  4 05:49:38 2014   Interval:2           Cswitch     240  Readch      190
                                                Syscall     130  Writech     400
CPU     User% Kern% Wait% Idle%   Physc  Entc%  Reads         1  Rawin         0
Total     0.0   0.0   0.0 100.0    0.01   0.07  Writes        1  Ttyout      188
                                                Forks         0  Igets         0
Network    BPS  I-Pkts  O-Pkts    B-In   B-Out  Execs         0  Namei         2
Total    3.68K   51.00    2.50   3.29K   396.0  Runqueue      0  Dirblk        0
                                                Waitqueue   0.0
Disk    Busy%      BPS     TPS  B-Read  B-Writ                   MEMORY
Total     0.0        0       0       0       0  PAGING           Real,MB   65792
                                                Faults        0  % Comp      8
FileSystem          BPS    TPS  B-Read  B-Writ  Steals        0  % Noncomp   0
Total             188.5   0.50   188.5       0  PgspIn        0  % Client    0
                                                PgspOut       0
Name           PID  CPU%  PgSp Owner            PageIn        0  PAGING SPACE
topas       6291672  0.0 3.51M root             PageOut       0  Size,MB   16896
reaffin      589854  0.0  640K root             Sios          0  % Used      0
swapper         524  0.0  448K root                              % Free    100
swapper         264  0.0  448K root             NFS (calls/sec)
getty       2491052  0.0  624K root             SerV2         0  WPAR Activ    0
java        3080656  0.0 78.8M root             CliV2         0  WPAR Total    0
sshd        6226168  0.0 1.23M root             SerV3         0  Press: "h"-help
gil         1769772  0.0  960K root             CliV3         0         "q"-quit

I even tried to check the bandwidth with a big file transfer and that looks fine.
Let me know if any other info is needed, actually we are stuck with this issue and we need to find a solution as soon as possible.
# 2  
Old 03-04-2014
Hi and welcome to this forum,

to me it sounds like a DNS Timeout / Network misconfiguration problem - but i am not sure!

I would check the resolv.conf, /etc/hosts, etc. In this case a ssh -vvv can help to debug the problem.

I had a similar problem with the ssh daemon, in this case the ipv6 name resolutions caused big delays.


Regards
# 3  
Old 03-04-2014
Hi,

this indeed sounds like a network configuration issue (dns). I saw this kind of behaviour before which was due to a dns misconfiguration.

Regards,

vincent
# 4  
Old 03-04-2014
Well, actually we are using internal IPs instead of names, so that dns config seems a little bit strange to me.
Anyway, I've done ssh -vvv and here is the result :
Code:
OpenSSH_5.2p1, OpenSSL 0.9.8x 10 May 2012
debug1: Reading configuration data //.ssh/config
debug1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Failed dlopen: /usr/krb5/lib/libkrb5.a(libkrb5.a.so):   0509-022 Cannot load module /usr/krb5/lib/libkrb5.a(libkrb5.a.so).
        0509-026 System error: A file or directory in the path name does not exist.

debug1: Error loading Kerberos, disabling Kerberos auth.
debug2: ssh_connect: needpriv 0

debug1: Connecting to 192.168.110.45 [192.168.110.45] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /.ssh/identity type -1
debug3: Not a RSA1 key file /.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /.ssh/id_rsa type 1
debug1: identity file /.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.2
debug1: match: OpenSSH_5.2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.2
debug2: fd 4 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: Entering the function :kex_choose_conf

debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 129/256
debug2: bits set: 497/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: filename /.ssh/known_hosts
debug3: check_host_in_hostfile: filename /etc/ssh/ssh_known_hosts
debug3: check_host_in_hostfile: filename /.ssh/known_hosts
debug3: check_host_in_hostfile: filename /etc/ssh/ssh_known_hosts
debug2: no key of type 0 for host 192.168.110.45
debug3: check_host_in_hostfile: filename /.ssh/known_hosts2
debug3: check_host_in_hostfile: filename /etc/ssh/ssh_known_hosts2
debug3: check_host_in_hostfile: filename /.ssh/known_hosts
debug3: check_host_in_hostfile: filename /etc/ssh/ssh_known_hosts
debug2: no key of type 2 for host 192.168.110.45
The authenticity of host '192.168.110.45 (192.168.110.45)' can't be established.
RSA key fingerprint is 93:c7:e8:e7:ce:31:42:82:1c:d8:c8:f5:2a:2d:79:57.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.110.45' (RSA) to the list of known hosts.
debug2: bits set: 546/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /.ssh/identity (0)
debug2: key: /.ssh/id_rsa (20084968)
debug2: key: /.ssh/id_dsa (0)
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: start over, passed a different list publickey,password,keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: The Key: 0

debug1: Trying private key: /.ssh/identity
debug3: no such identity: /.ssh/identity
debug1: After function load_identity_file

debug1: The Key: 1

debug1: Offering public key: /.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: The Key: 0

debug1: Trying private key: /.ssh/id_dsa
debug3: no such identity: /.ssh/id_dsa
debug1: After function load_identity_file

debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: userauth_kbdint: disable: no info_req_seen
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred:
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
root@192.168.110.45's password:

Also, /etc/resolv.conf is as follows :
Code:
nameserver 192.168.110.200

(the nameserver above is for our Oracle RAC)

Can that be because of IPv6? actually I disabled IPv6 by
Code:
chdev -l lo0 -a netaddr6=''

command
# 5  
Old 03-04-2014
At which point the SSH login hung? The Debug show no timestamps.

For a test with SSH vs. IPv6 you could enable the line ListenAddress 0.0.0.0 in the /etc/ssh/sshd_config and
restart the SSH-Daemon stopsrc -s sshd && startsrc -s sshd.

Regards
# 6  
Old 03-04-2014
Quote:
Originally Posted by aLuViAn
Well, actually we are using internal IPs instead of names, so that dns config seems a little bit strange to me.
It may be trying to do reverse lookups on the IPs.
# 7  
Old 03-04-2014
Quote:
Originally Posted by Corona688
It may be trying to do reverse lookups on the IPs.
How can I try that? Actually
Code:
127/8              127.0.0.1         U         1      9559 lo0      -      -

I have that in my routing table and

Code:
-bash-4.2# ifconfig -a
en4: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
        inet 192.168.110.45 netmask 0xffff0000 broadcast 192.168.255.255
         tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>
        inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
         tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

---------- Post updated at 10:11 PM ---------- Previous update was at 10:08 PM ----------

Quote:
Originally Posted by -=XrAy=-
At which point the SSH login hung? The Debug show no timestamps.
Here is when it hung:

Code:
debug1: Error loading Kerberos, disabling Kerberos auth.
debug2: ssh_connect: needpriv 0

and went on after a minute or so:
Code:
debug1: Connecting to 192.168.110.45 [192.168.110.45] port 22.

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

Java process showing high virtual memory

Hi All, I have a java process which is showing high virtual memory utilization in client server. But the same process is showing comparitively lesser virtual memory consumption. I understand that virtual memory shown is not of much importance for the general user in normal condition, my client... (2 Replies)
Discussion started by: mritusmoi
2 Replies

2. AIX

High Paging when lots of free memory AIX 5.3

I am new to AIX, I have few AIX 5.3 servers and I could see there are significant difference in paging space utilization on servers even though they are running same applications below server is working fine which shows 2-5 % paging usage throuh out the day cpu_scale_memp = 8... (12 Replies)
Discussion started by: bibish
12 Replies

3. HP-UX

Find out most memory consumption process in HP UX

Dear All, I want to find out top 15 memory consumption processes in HP UX. Can anyone give me any idea about it? Kauser (2 Replies)
Discussion started by: makauser
2 Replies

4. Solaris

[DOUBT] Memory high in idle process on Solaris 10 (Memory Utilization > 90%)

Hi Experts, Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing. Why memory utilization on solaris always looks high? I have statement about memory on solaris, is this true: Memory in solaris is used for... (4 Replies)
Discussion started by: edydsuranta
4 Replies

5. AIX

AIX memory usage always high

hi, I want to ask , my AIX 6.1 is always used about 97% memory. Is this normal ? or any command can free up memory like Linux ? thanks. (1 Reply)
Discussion started by: virusxx
1 Replies

6. AIX

AIX resources consumption group by process pattern

Hi, I have AIX processes something as below, how to know the resources consumption group by process pattern "price" and "devdb"? eg: CPU RAM WIO price 20% 250M 5% devdb 30% 1000M 8% oracle 2990122 1 0 10:33:39 - 0:00... (1 Reply)
Discussion started by: victorcheung
1 Replies

7. AIX

Memory consumption issue in AIX box

Hi, monitor memory usage on AIX machine on any day of the week from 3:00 PM ET to 5:00 PM ET - Povide min and max memory consumption. Determine if all of the available memory is visible to the operating system. If it is not, determine the amount of memory which may be allocated to the... (3 Replies)
Discussion started by: Rookie_newbie
3 Replies

8. UNIX for Dummies Questions & Answers

Slow printing / CUPS - process "parallel" high cpu

Hello. I have a slackware system running cups with an HP laserJet 2100 connected via parallel port in ECP mode. Print jobs are working. Very slowly. 15K test print out of cups takes about 2 minutes to complete. When the printout is on the way to the printer, the process "parallel" uses... (0 Replies)
Discussion started by: agentrnge
0 Replies

9. Solaris

real memory consumption of a process

hi pmap, prstat and ps gives the RSS which is shared memory. I need to know the consumption of real memory. how to do that? thx (4 Replies)
Discussion started by: melanie_pfefer
4 Replies

10. AIX

High memory usage in AIX 5.1

Hi, We have AIX 5.1 machine of RAM 8 GB and paging space is 8GB. we are getting high memory usage of almost 99%.Can anybody please help in this ? Partial vmstat o/p kthr memory ----- ----------- r b avm fre 2 1 278727 1143 There is no paging issue.Becoz in... (5 Replies)
Discussion started by: jayakumarrt
5 Replies
Login or Register to Ask a Question