Login or Register to Ask a Question and Join Our Community


AIX

High Paging when lots of free memory AIX 5.3

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems AIX High Paging when lots of free memory AIX 5.3
# 1  
Old 08-27-2014
High Paging when lots of free memory AIX 5.3
I am new to AIX, I have few AIX 5.3 servers and I could see there are significant difference in paging space utilization on servers even though they are running same applications

below server is working fine which shows 2-5 % paging usage throuh out the day

Code:
cpu_scale_memp = 8
data_stagger_interval = 161
defps = 1
force_relalias_lite = 0
framesets = 2
htabscale = n/a
kernel_heap_psize = 4096
kernel_psize = 16777216
large_page_heap_size = 0
lgpg_regions = 0
lgpg_size = 0
low_ps_handling = 1
lru_file_repage = 0
lru_poll_interval = 10
lrubucket = 131072
maxclient% = 90
maxfree = 1088
maxperm = 7222844
maxperm% = 90
maxpin = 6766853
maxpin% = 80
mbuf_heap_psize = 65536
memory_affinity = 1
memory_frames = 8388608
mempools = 4
minfree = 960
minperm = 240760
minperm% = 3
pinnable_frames = 6739105
psm_timeout_interval = 5000
pta_balance_threshold = n/a
relalias_percentage = 0
rpgclean = 0
rpgcontrol = 2
scrub = 0
scrubclean = 0
soft_min_lgpgs_vmpool = 0
spec_dataseg_int = 512
strict_maxclient = 1
strict_maxperm = 0
v_pinshm = 0
vm_modlist_threshold = -1
vmm_fork_policy = 1
vmm_mpsize_support = 1
wlm_memlimit_nonpg = 1

we have another server in which the paging utilization keep incresing and we have to restart the application everyday to make it normal Smilie

here is the tunable parameters


Code:
cpu_scale_memp = 8
data_stagger_interval = 161
defps = 1
force_relalias_lite = 0
framesets = 2
htabscale = n/a
kernel_heap_psize = 4096
kernel_psize = 16777216
large_page_heap_size = 0
lgpg_regions = 0
lgpg_size = 0
low_ps_handling = 1
lru_file_repage = 0
lru_poll_interval = 10
lrubucket = 131072
maxclient% = 50
maxfree = 1088
maxperm = 4524244
maxperm% = 50
maxpin = 7611036
maxpin% = 80
mbuf_heap_psize = 65536
memory_affinity = 1
memory_frames = 9437184
mempools = 5
minfree = 960
minperm = 1809696
minperm% = 20
pinnable_frames = 7640488
psm_timeout_interval = 20000
pta_balance_threshold = n/a
relalias_percentage = 0
rpgclean = 0
rpgcontrol = 2
scrub = 0
scrubclean = 0
soft_min_lgpgs_vmpool = 0
spec_dataseg_int = 512
strict_maxclient = 1
strict_maxperm = 0
v_pinshm = 0
vm_modlist_threshold = -1
vmm_fork_policy = 1
vmm_mpsize_support = 1
wlm_memlimit_nonpg = 1

let me know if requre any more details to find the issue

Bibi
High Paging when lots of free memory AIX 5.3-cio1png
Last edited by Scrutinizer; 08-27-2014 at 07:32 AM.. Reason: CODE tags
# 2  
Old 08-27-2014
Hi,

out of the blue I would try to set minperm% to 5 and maxperm% to 90 (maxclient% as well to 90) and check if the behaviour changes.
lru_file_repage is already set to 0 so you usually go with something above.

What type of application(s) is running on the box, a DB, ...?

It would also be interessting to see which processes are using the most paging space:
Code:
svmon -P -O sortentity=pgsp

Can you also please post the outputs of:
Code:
vmstat -w -t 1 10          # When you notice paging
vmstat -vs                 # This one anytime you want

And the most important of all:
Use code tags for the output Smilie
# 3  
Old 08-27-2014
Thanks for the reply and sorry for not using the code tag.
This server is running java/weblogic application
Code:
$ vmstat -w -t 1 10
System configuration: lcpu=24 mem=36864MB ent=5.20
 kthr          memory                         page                       faults                 cpu             time
------- --------------------- ------------------------------------ ------------------ ----------------------- --------
  r   b        avm        fre    re    pi    po    fr     sr    cy    in     sy    cs us sy id wa    pc    ec hr mi se
  9   0    8514838       9984     0     6     9   514   1866     0  2285  12154  3000 72  2 27  0  3.83  73.6 15:48:23
  3   0    8514838      11198     0     6     3  1288   4091     0   771   9390  2266 41  2 57  0  2.23  43.0 15:48:24
  9   0    8514852      10921     0    35    12    19    109     0  3209  16445  5013 77  2 20  0  4.15  79.8 15:48:25
  4   0    8514853      10525     0     8     4     9     42     0  1789  14661  3543 72  2 26  0  3.88  74.7 15:48:26
  5   1    8514853      10049     0    21     7     9     69     0  2699  28619  5436 87  3 10  0  4.70  90.5 15:48:27
  6   0    8514868      10014     0    10    10   276    851     0  2631  23858  6503 90  2  7  0  4.83  93.0 15:48:28
  3   0    8514871      11310     0    17    16  1674   3382     0  2399  23227  4980 73  3 23  0  4.00  77.0 15:48:29
  6   0    8514873      11132     0    10     0     9     43     0  2256  17267  4099 76  3 21  0  4.10  78.9 15:48:30
  3   0    8514875      10865     0    22     3     9     53     0  2190  20454  4908 76  2 22  0  4.06  78.1 15:48:31
  3   0    8514877      10337     0     6     0     0      0     0  2645  14323  4269 69  2 28  0  3.75  72.1 15:48:32

Code:
$ vmstat -vs
          26718300070 total address trans. faults
           4324247173 page ins
           3782710509 page outs
             82381547 paging space page ins
             76413162 paging space page outs
                    0 total reclaims
           8121975552 zero filled pages faults
             11657654 executable filled pages faults
          15016368991 pages examined by clock
                25189 revolutions of the clock hand
           5277096998 pages freed by the clock
             65041712 backtracks
               171316 free frame waits
                    0 extend XPT waits
            281450154 pending I/O waits
           8082022155 start I/Os
            846120866 iodones
          42694017510 cpu context switches
          16834766912 device interrupts
           1952030653 software interrupts
          19827299303 decrementer interrupts
            165373147 mpc-sent interrupts
            165373098 mpc-received interrupts
           1330553110 phantom interrupts
                    0 traps
         190748815496 syscalls
              9437184 memory pages
              9048488 lruable pages
                11461 free pages
                    5 memory pools
              1796762 pinned pages
                 80.0 maxpin percentage
                 20.0 minperm percentage
                 50.0 maxperm percentage
                 19.3 numperm percentage
              1753772 file pages
                  0.0 compressed percentage
                    0 compressed pages
                 19.3 numclient percentage
                 50.0 maxclient percentage
              1753772 client pages
                    0 remote pageouts scheduled
                 4760 pending disk I/Os blocked with no pbuf
             14981367 paging space I/Os blocked with no psbuf
                 1972 filesystem I/Os blocked with no fsbuf
                27374 client filesystem I/Os blocked with no fsbuf
               409645 external pager filesystem I/Os blocked with no fsbuf
                    0 Virtualized Partition Memory Page Faults
                 0.00 Time resolving virtualized partition memory page faults

Code:
$ svmon -P -O sortentity=pgsp
Unit: page

Code:
Total Paging Space   Percent Used
      30720MB              24%

# 4  
Old 08-27-2014
Lots of free memory on a well warmed up server is just an indication of lots of exiting processes as part of the app. Exit frees the RAM footprinnt of the local parts of the process (heap and stack -- code is usually shared pages), bumping up free RAM until paging in eats it away. Sometimes this indicates too much shell programming, or bad shell programming, as processes are executed and discarded over and over. Sometimes these are spun off from server functions. The JAVA webserver process is probably not really freeing RAM to the OS, as free() is just putting it back in the free pool of the memory allocation arena awaiting the next malloc(), or the JAVA gc equivalent.
# 5  
Old 08-28-2014
Thanks for the data.

The svmon did not show much, seems this sort is an > AIX 5.3 thing; try this one and check which has/have by far the highest value in the column "Pgsp":
Code:
svmon -P | awk '/Pid C/ {h=$0; next} NF == 9 && /^[0-9]/ {if($5 != 0) l[c++]=$0} END{print h; for(x=0; x<c; x++){print l[x]} }'

I would set parameters as recommended in my former post and see if the paging stops, hopefully.
This may take some time since there is still a lot out on the disk that needs at least to be paged in when it's needed. To speed this up, reboot the box after setting the following command...
You can set them online with the following command:
Code:
vmo -p -o minperm%=5 -o maxclient%=90 -o maxperm%=90

The -p makes it permanent and survives a boot.
This User Gave Thanks to zaxxon For This Post:
bibish
# 6  
Old 08-28-2014
Hi,

I've seen this type of thing before although it was quite a number of years ago, it actually turned out to be a memory leak from badly written code. It was hard to track it down, we had four P695's alll running the exact same application suite - just turned out to be that the one used for reporting had the problem.

So possible looking at the usage of eack of the boxes may provide a pointer.

Regards

Dave
# 7  
Old 08-28-2014
Looking for leaks, find processes with high and growing total VM using 'ps -o vsz'. (This is on an hp-ux, but ps should work similarly on AIX.):
Code:
$ (export UNIX95=true ; ps -exo 'pid,sz,vsz,args')|(line;sort -nr +2 -3)|pg
  PID   SZ     VSZ COMMAND
 8081 18275   74692 mad -u root -g bin
15976    0   52124 xterm -T a-5 -n 5 -geometry 80x25 -fn 12x24 -sb -sl 99999 -vb -bg white -fg black
 1878 12576   51328 /appl/banktools/APPQcime/jre/bin/PA_RISC/java -Dprogram.name=../tools/start . . . org.tanukisoftware.wrapper.WrapperSimpleApp com.appiq.cxws.main.HpMain
 1683 5938   24052 /opt/perf/bin/scopeux
 1856 7096   22784 ./EpicCore
 4602 6810   21632 ./EpicShadow
 .
 .
 .
 .

These 2 Users Gave Thanks to DGPickett For This Post:
bibish rbatte1
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. AIX

AIX 7.1 on Power 750 is very slow with no high memory or process consumption

Hi, This thread has been posted before on linuxquestions.org, but no answer, maybe because this is unix question and not linux. I'm posting the same thread here, hope I can get an answer from someone in the meantime, I wish I could post of emergency thread but it needs bits which I don't have :... (6 Replies)
Discussion started by: aLuViAn
6 Replies

2. AIX

How much total and free memory I have in my aix 5.3 server?

good morning, how I can know how much total and free memory I have in my AIX 5.3 server, and this is shown in megabytes or gigabytes? Thank you very much. (4 Replies)
Discussion started by: systemoper
4 Replies

3. AIX

How to find AIX Free Memory?

All, AIX: 6.1 64 bits How to find out Free memory available on AIX 6.1 64 bits When I used : svmon -G size inuse free pin virtual mmode memory 1048576 612109 191151 215969 549824 Ded-E pg space 4325376 ... (1 Reply)
Discussion started by: a1_win
1 Replies

4. AIX

AIX memory usage always high

hi, I want to ask , my AIX 6.1 is always used about 97% memory. Is this normal ? or any command can free up memory like Linux ? thanks. (1 Reply)
Discussion started by: virusxx
1 Replies

5. AIX

Lots of page faults and free memory

Hello, I've been reading your forums for quite a while and the great amount of information I find here always come in hand.This time however, I need some specific help... I have a doubt with an AIX server which I'm failing to understand as I'm new to its concept of memory management... ... (8 Replies)
Discussion started by: flpgdt
8 Replies

6. Programming

How to deal with lots of data in memory in order not to run out of memory

Hi, I'm trying to learn how to manage memory when I have to deal with lots of data. Basically I'm indexing a huge file (5GB, but it can be bigger), by creating tables that holds offset <-> startOfSomeData information. Currently I'm mapping the whole file at once (yep!) but of course the... (1 Reply)
Discussion started by: emitrax
1 Replies

7. AIX

Free Memory in aix

Hi, I want to know how to find out free physical memory in aix, (11 Replies)
Discussion started by: manoj.solaris
11 Replies

8. AIX

High memory usage in AIX 5.1

Hi, We have AIX 5.1 machine of RAM 8 GB and paging space is 8GB. we are getting high memory usage of almost 99%.Can anybody please help in this ? Partial vmstat o/p kthr memory ----- ----------- r b avm fre 2 1 278727 1143 There is no paging issue.Becoz in... (5 Replies)
Discussion started by: jayakumarrt
5 Replies

9. AIX

How to check free/total Memory in AIX

Friends , i have a question how to check the total memomry and free memory in AIX, We have vmstat ,svmon and topas commands.Which command among the will give the true figure. (6 Replies)
Discussion started by: vimalbhan
6 Replies

10. UNIX for Advanced & Expert Users

Shared memory shortage but lots of unused memory

I am running HP-UX B.11.11. I'm increasing a parameter for a database engine so that it uses more memory to buffer the disk drive (to speed up performance). I have over 5GB of memory not being used. But when I try to start the DB with the increased buffer parameter I get told. "Not... (1 Reply)
Discussion started by: cjcamaro
1 Replies
Login or Register to Ask a Question