AIX prompt delayed when users increase

 
Thread Tools Search this Thread
Homework and Emergencies Emergency UNIX and Linux Support AIX prompt delayed when users increase
# 1  
Old 02-02-2017
AIX prompt delayed when users increase

I am facing a situation where when there are lesser users, i am able to login to the AIX server. If the number of users increase - the login prompt is getting delayed. Sometimes even timeout occurs. This is after the upgrade to AIX 7.1 TL 4. Can someone suggest a way to overcome this situation?
# 2  
Old 02-02-2017
How are you logging in?

What is the number of users on the system when you can no longer login reliably?

What is the load average on the system when you can no longer login reliably?

Has the amount of work assigned to this system changed since the upgrade? If so, how?
# 3  
Old 02-02-2017
Quote:
Originally Posted by ggayathri
Can someone suggest a way to overcome this situation?
Maybe, but to do so we need to know more about the system:

How many is an "increased number" of users? (no exact number required, but are we talking about 10, 100 or 1000 users here?)

What is the size of the system? How many processors? How much memory? Have you tried a vmstat -tw 1 in a time of many users? also a vmstat -vs might help. If so, please post significant portions of the outpput here (within CODE-tags).

How is the name resolution done? We had "network problems", where the network was working fine but the name server took a long time to answer so that the connection seemed to be made slowly. Overall, can you describe you network a bit (in case the culprit is there)?

How is your system used by the many users you describe? Are they using a certain application, working on the command line mainly, do they only connect via the application layer (like a database running on the server and your users use DB Visualizer or a similar tool to connect to it)?

This is - off the top of my head - all i can come up with. Please provide a few more data and we take it from there.

I hope this helps.

bakunin

/PS: seems like i am not typing fast enough to catch up with Don. ;-)
# 4  
Old 02-03-2017
The number of users when this issue was happening was 250. Connection is using SSH. We connect using any of the GUI (xterm) to login to the servers.
Network does not seem to be an issue.
Code:
System configuration: lcpu=64 mem=256511MB

 kthr          memory                         page                       faults           cpu       time
------- --------------------- ------------------------------------ ------------------ ----------- --------
  r   b        avm        fre    re    pi    po    fr     sr    cy    in     sy    cs us sy id wa hr mi se
  0   0    4749214    4687647     0     0     0     0      0     0     6   2970   760  0  0 99  0 00:19:28
  0   0    4750341    4686520     0     0     0     0      0     0    18  35490   893  0  1 99  0 00:19:29
  0   0    4749218    4687643     0     0     0     0      0     0     2  14121   815  0  0 99  0 00:19:30
  0   0    4749218    4687638     0     0     0     0      0     0    17   3871   836  0  0 99  0 00:19:31


Code:
vmstat -vs
           4465310913 total address trans. faults
            294360283 page ins
             49324316 page outs
                    0 paging space page ins
                    0 paging space page outs
                    0 total reclaims
           1465526223 zero filled pages faults
                60683 executable filled pages faults
            114267450 pages examined by clock
                    0 revolutions of the clock hand
             55792932 pages freed by the clock
             13653791 backtracks
                    0 free frame waits
                    0 extend XPT waits
             52515800 pending I/O waits
            343680275 start I/Os
             80649413 iodones
           3850688645 cpu context switches
            228964899 device interrupts
             44816910 software interrupts
           2057467216 decrementer interrupts
               505680 mpc-sent interrupts
               505680 mpc-receive interrupts
               230520 phantom interrupts
                    0 traps
         262745045277 syscalls
             65666912 memory pages
             63779457 lruable pages
              4687825 free pages
                    8 memory pools
              4365673 pinned pages
                 90.0 maxpin percentage
                  3.0 minperm percentage
                 90.0 maxperm percentage
                 88.0 numperm percentage
             56179798 file pages
                  0.0 compressed percentage
                    0 compressed pages
                 88.0 numclient percentage
                 90.0 maxclient percentage
             56171829 client pages
                    0 remote pageouts scheduled
                 1190 pending disk I/Os blocked with no pbuf
                    0 paging space I/Os blocked with no psbuf
                 2400 filesystem I/Os blocked with no fsbuf
                44434 client filesystem I/Os blocked with no fsbuf
                25130 external pager filesystem I/Os blocked with no fsbuf
                  7.3 percentage of memory used for computational pages


Moderator's Comments:
Mod Comment Changed icode to code tags.


---------- Post updated 02-03-17 at 07:17 AM ---------- Previous update was 02-02-17 at 09:20 PM ----------

Right now, when the problem is happening

Code:
vmstat -vs
           4704754850 total address trans. faults
            295484675 page ins
             50773171 page outs
                    0 paging space page ins
                    0 paging space page outs
                    0 total reclaims
           1548595741 zero filled pages faults
                62036 executable filled pages faults
            114267450 pages examined by clock
                    0 revolutions of the clock hand
             55792932 pages freed by the clock
             14409311 backtracks
                    0 free frame waits
                    0 extend XPT waits
             52975790 pending I/O waits
            346252431 start I/Os
             81917039 iodones
           3938511790 cpu context switches
            233154812 device interrupts
             46845701 software interrupts
           2125584724 decrementer interrupts
               528400 mpc-sent interrupts
               528400 mpc-receive interrupts
               242521 phantom interrupts
                    0 traps
         264369806068 syscalls
             65666912 memory pages
             63779457 lruable pages
              3413598 free pages
                    8 memory pools
              4381020 pinned pages
                 90.0 maxpin percentage
                  3.0 minperm percentage
                 90.0 maxperm percentage
                 89.8 numperm percentage
             57312540 file pages
                  0.0 compressed percentage
                    0 compressed pages
                 89.8 numclient percentage
                 90.0 maxclient percentage
             57304175 client pages
                    0 remote pageouts scheduled
                 1235 pending disk I/Os blocked with no pbuf
                    0 paging space I/Os blocked with no psbuf
                 2400 filesystem I/Os blocked with no fsbuf
                44434 client filesystem I/Os blocked with no fsbuf
                25136 external pager filesystem I/Os blocked with no fsbuf
                  7.5 percentage of memory used for computational pages


Last edited by Corona688; 02-03-2017 at 11:21 AM.. Reason: Fixed broken icode again
# 5  
Old 02-03-2017
Please use code tags, not icode tags, press the Image button.
# 6  
Old 02-08-2017
A debugging tip. You can trace the sshd.
Code:
# proctree | grep -w sshd
   258180   /usr/sbin/sshd a
      274564   sshd: root@pts/0 a

Then trace the originator/listener i.e. pid 258180
Code:
# truss -f -p 258180

And then do a login. You'll see what it does, and where it takes a long time.
These 2 Users Gave Thanks to MadeInGermany For This Post:
# 7  
Old 03-19-2017
Further investigation has shown the delays are occurring when the loginsuccess() function, called by sshd, tries to acquire a lock on the file /etc/security/lastlog.
Any idea what could be causing this? Or where to go look for a solution
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. AIX

Increase filesystem size AIX 5.3

I an trying to increase the file size on an AIX 5.3 box. I think i am missing the correct syntax for the command. Here is was i am trying on a test box # lsvg rootvg VOLUME GROUP: rootvg VG IDENTIFIER: 0000bd8b00004c00000 0010d8ed7a76e VG STATE: active ... (3 Replies)
Discussion started by: fierfek
3 Replies

2. AIX

AIX 4.2 increase ethernet speed

Hello. I have a server RS/6000 with AIX 4.2. The server have a network card with 10/100mbps speed, but for some reason, the ethernet only runs at 10Mbps, it doesn,t matter if I put the net cable on a 100Mbps switch, it keeps running at 10Mbps speed.... I know how to view/change ethernet speed... (1 Reply)
Discussion started by: little_ball
1 Replies

3. AIX

how many su - users I ejecute in prompt AIX 5.3

Hi Colleagues, My English is pretty bad, the help I need is this: entered into a machine with OS AIX 5.3 and i execute su - user1 and then i execute su - user2 and then i execute su - user3 and finally its su - user4. there is some command that tells me the its that i have made? Thank you... (2 Replies)
Discussion started by: systemoper
2 Replies

4. AIX

Increase LUN size in AIX with VIOS and HACMP

Hello! I have this infraestructure: - 1 POWER7 with single VIOS on Site A. - 1 POWER6 with single VIOS on Site B. - 1 LPAR called NodeA as primary node for PowerHA 6.1 on Site A. - 1 LPAR called NodeB as secondary (cold) node for PowerHA 6.1 on SiteB. - 1 Storage DS4700 on Site A. - 1... (8 Replies)
Discussion started by: enzote
8 Replies

5. AIX

ssh keys - no password prompt from AIX to RPA

hello, i am running an AIX 5.3 machine and i want to connect via ssh to the RPA Management site without prompting for password. i already had a public key of this server as i use the same thing for ssh connection with other AIX machines. i connected to the RPA Management Site and i run the... (9 Replies)
Discussion started by: omonoiatis9
9 Replies

6. UNIX for Dummies Questions & Answers

scp not working because of prompt (AIX, tcsh)

Hello, I have this problem: I have a server to which I ssh, and it has a special prompt request. The prompt is done by a ?prompt command. It is fine with SSH, since the prompt I guess gets some input, but when I use SCP, the copy always fails. So, I was wondering if there is maybe a... (1 Reply)
Discussion started by: lastZenMaster
1 Replies

7. AIX

Timestamp in AIX Prompt

I am using ksh under IBM's AIX, version 6.1. I wopuld like to add the current time to my prompt. I understand that I can change the value of $PS1 in my .profile. So far my attempts at setting the time have resulted in setting the time to what it was when the .profile was executed. ( It doesn't... (2 Replies)
Discussion started by: PeterF
2 Replies

8. AIX

Custom AIX Prompt

In my .profile, my prompt is set like this: set -o vi PS1=`logname`@`hostname -s`:'$PWD>' Is there a way to show what the history number would be of the command I'm typing in the prompt? For example, I frequently run commands then run 'history' to pull up the history number of a command... (2 Replies)
Discussion started by: ptrotter
2 Replies

9. AIX

how do I increase the storage space I am new to AIX

Host Name - xxxxxxx IP Address - xxxxxxxxx Alert Msg - The percentage of available storage space (DMXMemory) is low (49.54374442289481 percent). Time received - 14:23 Time Logged - 14:55 Suggested Group - MR-UNIX (5 Replies)
Discussion started by: Nyiko
5 Replies
Login or Register to Ask a Question