Login or Register to Ask a Question and Join Our Community


AIX

type of pages being paged in/out

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems AIX type of pages being paged in/out
# 8  
Old 03-04-2011
Hi,

from this output, I would only recommend to add 20 GB memory and your paging will stop. From your stats you are using close to 100% memory computational - no wonder that your box is paging - and yes this will for sure be DB content as well ... and if your DBAs are doing rman backups on top it becomes real bad.

What you could try is mount your oracle filesystems with noatime option and switch oracle to setall - this will give you more free memory into your free list and may reduce your memory footprint.

Regards
zxmaus
# 9  
Old 03-04-2011
zxmaus, thanks for your suggestions.

how did you calculate that system needs 20Gb?

i found in one IBM book, that memory needs are virtual + pers + clnt from svmon output.

so the deficit was 19496715 + 5 + 6447378 - 20971520 = 4972578 * 4Kb = 19890312 Kb = 19Gb

are these calculations correct?

Code:
# svmon -G
               size       inuse        free         pin     virtual   mmode
memory     20971520    20895377       10607     4564324    19496715     Ded
pg space   16842752     5667782

               work        pers        clnt       other
pin         3952074           0           0      612250
in use     14447994           5     6447378

PageSize   PoolSize       inuse        pgsp         pin     virtual
s    4 KB         -    10584481     5480166      811284     9154139
m   64 KB         -      644431       11726      234565      646411

i don't understand yet what is "virtual" represent in svmon. the value 19496715 is greater than paging space and lower than real memory. can you explain this?
i can show you some visual graphs captured by nmon for the last 3 days. we are interested in the time from 8 to 17 o'clock

this from last Tuesday when system was paging much
type of pages being paged in/out-01032011gif

this is from Wednesday, the system paged out only until 12, and right after that the percentage of computational processes memory increased to 80%
type of pages being paged in/out-02032011gif

this is from Thursday, when the system almos did not page from 08 till 17 and as you see all this time computational memory was on the level of 75%-80%
type of pages being paged in/out-03032011gif


it seems to me that system is comfortable when computational memory consumes 80% of the total memory, which is 80Gb. 20% is used by the system, so application needs are 60% of 80Gb which is 48Gb.
# 10  
Old 03-04-2011
I have a few hundred oracle boxes - in my experience the systems are most comfortable when comp (the avm value in vmstat x 4k) doesnt exceed 80% as this leaves enough memory for all the oracle forked processes, IO buffering, batch processing and so on.
When my memory utilization exceeds these 80% than my system starts scanning / freeing memory which utilizes cpu and slows down the DB as the system waits for sufficient freed up memory to continue processing - which obviously is bad. The higher the scan to free ratio - so the more pages need to be scanned to free up the memory I actually need for the given workload - the slower the system gets and the more cpu is utilized. So I make sure I always have plenty of memory - as particularly for oracle the need of non-comp memory is very valid as its usually a filesystem based DB - and not finding filecache if needed slows down the DB too as no IO can happen ...
Please note - during rman backups you still will see some scan and free as this puts - at least in my environments - a large amount of additional load onto the systems. So my 80% are during busy times but not when rman runs. Nmon is pretty helpful to find out what is good for your system and when you do have your busy times.

Virtual memory btw is physical memory + pagingspace in 4k pages. Virtual memory in use is how much of this you are actively using - ideally visibly less than you physically have Smilie

Regards
zxmaus
# 11  
Old 03-06-2011
Quote:
# vmstat -v
20971520 memory pages
20286310 lruable pages
10525 free pages
4 memory pools
4631080 pinned pages
80.0 maxpin percentage
5.0 minperm percentage
95.0 maxperm percentage
31.3 numperm percentage
6357498 file pages
0.0 compressed percentage
0 compressed pages
31.3 numclient percentage
95.0 maxclient percentage
6357493 client pages
0 remote pageouts scheduled
1167074 pending disk I/Os blocked with no pbuf
600159 paging space I/Os blocked with no psbuf
2484 filesystem I/Os blocked with no fsbuf
443 client filesystem I/Os blocked with no fsbuf
249209795 external pager filesystem I/Os blocked with no fsbuf
Isn't this almost 25GB FS data in physical memory while having >20GB in the paging?
Reducing (forcing down) FS caching would be one way to go in my opinion, why an Oracle box needs to cache 25GB FS data?
# 12  
Old 03-06-2011
oh many reasons:
- because oracle is a filesystem + process based DB
- because each oracle forked process needs memory - I assume this server has thousands of processes
- because every batch / backup / DB load requires lots of file caching on top of what is in the SGA - each IO will use 1 page (4 k) of memory
# 13  
Old 03-08-2011
On thing that can run paging up unnecessarily is static linking of apps, something you can change for apps developed locally. All apps using libs dynamically linked are using the same pages, not copies, and since those pages are more frequently referenced, they stay in RAM.

We used to get a lot of RDBMS out of a small platform by designing batch processes to process N records at a time and then commit. We also found that over-use of updatable cursors increased processing. The way Oracle works, a long select can end up owning many pages as other processes update or delete those rows. So, it helps the whole system to do things in small batches, and even in select programs, a commit may release pages tied up by update-capable cursors. If you think of it, even processing 128 records per commit, you have 99+% of any economy of scale over one at a time. Any locks are released sooner, so interactive can get access. As batches get smaller, working set pages are in RAM or CPU cache more often, and finished pages can roll out and not soon return. Smaller batches also are more likely not to overwhelm cachng and buffering in disk subsystems, slowing I/O to media speed. Also, the system tuning does not change on more active days, just the batch run time.

Interactive row sets tend to be small, but batch can bring a lot of pages, an unpredicatable number, into play at once.
These 2 Users Gave Thanks to DGPickett For This Post:
zaxxon zxmaus
# 14  
Old 03-09-2011
Thanks for your advices, guys.

I don't have direct access to Oracle DB and don't know what it's doing. DB is managed by our client.

At this moment system is not paging out, but there are some issues anyway. I see many waits in the queue and some blocked I/O operations.
See below.

Code:
# vmstat -Iwt 1 20

System configuration: lcpu=12 mem=81920MB ent=4.15

   kthr     memory                 page                       faults                 cpu             time
-----------  ------------------------------------ ------------------ ----------------------- --------
  r   b   p        avm        fre    fi    fo    pi    po    fr     sr    in     sy    cs us sy id wa    pc    ec hr mi se
 20   8   0   19073983      15268  9364  1896     0     0  8521  24873  3872  49377 18149 79 21  0  1  5.42 130.5 11:17:19
  7  16   0   19065828      12586 10785  1191     5     0  1129   2890  3847  37785 15754 91  9  0  1  4.91 118.4 11:17:20
  5  11   0   19066437      11007  5477   147    31     0  4516  10454  3337  39842 13525 92  7  0  1  4.70 113.3 11:17:21
  4  15   0   19066152      10584  7502  2537    35     0  9303  19316  4053  23820 20048 89 10  0  1  4.67 112.5 11:17:22
  7  13   0   19069319      10611  9314  1618     1     0 13935  35082  3913  30277 23417 87 11  0  1  4.73 113.9 11:17:23
  7   9   0   19066310      14833  9568    79    82     0 10966  19981  4539  21673 20493 88 10  0  1  4.51 108.6 11:17:24
  5  11   0   19051653      20545 10865  2252   154     0  4197   5532  5330  26891 21889 86 13  0  1  4.60 110.9 11:17:25
  7  12   0   19052092      11591 11976  1283   628     0  8389  11622  6363  47647 33071 86 13  0  1  5.00 120.4 11:17:26
  6  11   0   19050240      10796 16977    97     0     0 14361  42973  5563  34192 32122 81 17  0  2  4.62 111.3 11:17:27
  8  11   0   19049775      10704 14122  2296     0     0 16544  18781  4727  17908 28365 85 14  0  1  4.71 113.4 11:17:28
  3  16   0   19049791      10758 16861  1399     0     0 17468  20331  5714  23736 29753 81 17  0  2  4.43 106.7 11:17:29
  6   9   0   19049410      10842 13262    28     0     0 13524  15228  5683  27415 25719 75 16  2  7  3.84  92.5 11:17:30
  7  18   0   19049421      10776  3159  2324     0     0  4760   5388  1846  18286  9474 77  7  3 13  3.53  85.1 11:17:31
  5  19   0   19049422      10637  3774   634     1     0  4220   4854  3565  26489 11654 80  7  6  7  3.66  88.1 11:17:32
  0   0   0   19049423      10692  3165     8     0     0  3228   3634  2831  19911  8719 79  6  6  9  3.58  86.3 11:17:33
  7  14   0   19049692      10613 12350   971     0     0 13574  15363  4841  32162 25576 85 14  0  1  4.29 103.3 11:17:34
 10  10   0   19049628      10804  9855  2179     0     0 11898  13431  3764  12747 22990 83 14  0  2  4.10  98.8 11:17:35
  6  16   0   19048700      11440  7508   152     0     0  7440   8292  4032  35291 25988 86 12  0  2  4.44 107.0 11:17:36
  6  18   0   19054224      10914  9375  3100     0     0 17225  20303  4621  31231 31954 87 12  0  1  5.10 122.9 11:17:37
 14  11   0   19058459      10781 14799  9437     0     0 28163  32285  5057  25853 31991 88 11  0  1  5.89 141.9 11:17:38

# vmstat -v
             20971520 memory pages
             20286310 lruable pages
                32162 free pages
                    4 memory pools
              4698583 pinned pages
                 80.0 maxpin percentage
                  5.0 minperm percentage
                 95.0 maxperm percentage
                 18.3 numperm percentage
              3714940 file pages
                  0.0 compressed percentage
                    0 compressed pages
                 18.3 numclient percentage
                 95.0 maxclient percentage
              3714935 client pages
                    0 remote pageouts scheduled
              1242466 pending disk I/Os blocked with no pbuf
               820283 paging space I/Os blocked with no psbuf
                 2484 filesystem I/Os blocked with no fsbuf
                  443 client filesystem I/Os blocked with no fsbuf
            257814289 external pager filesystem I/Os blocked with no fsbuf

and the number of "external pager filesystem I/Os blocked with no fsbuf" is growing every minute with 20-30 blocked operations.

Code:
# while true; do date; vmstat -v | grep external; sleep 10; done
Wed  9 Mar 11:19:24 2011
            257815036 external pager filesystem I/Os blocked with no fsbuf
Wed  9 Mar 11:19:34 2011
            257815036 external pager filesystem I/Os blocked with no fsbuf
Wed  9 Mar 11:19:44 2011
            257815036 external pager filesystem I/Os blocked with no fsbuf
Wed  9 Mar 11:19:54 2011
            257815036 external pager filesystem I/Os blocked with no fsbuf
Wed  9 Mar 11:20:04 2011
            257815036 external pager filesystem I/Os blocked with no fsbuf
Wed  9 Mar 11:20:14 2011
            257815045 external pager filesystem I/Os blocked with no fsbuf
Wed  9 Mar 11:20:24 2011
            257815058 external pager filesystem I/Os blocked with no fsbuf
Wed  9 Mar 11:20:34 2011
            257815058 external pager filesystem I/Os blocked with no fsbuf
Wed  9 Mar 11:20:44 2011
            257815087 external pager filesystem I/Os blocked with no fsbuf
Wed  9 Mar 11:20:54 2011
            257815087 external pager filesystem I/Os blocked with no fsbuf
Wed  9 Mar 11:21:04 2011
            257815087 external pager filesystem I/Os blocked with no fsbuf
Wed  9 Mar 11:21:14 2011
            257815087 external pager filesystem I/Os blocked with no fsbuf
Wed  9 Mar 11:21:24 2011
            257815087 external pager filesystem I/Os blocked with no fsbuf

i'm going to increase the value of j2_dynamicBufferPreallocation, which is equal to 16 now.
Can you suggest, how to determine what should i set for j2_dynamicBufferPreallocation ?
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

more f skips two pages

Hi, I am new to linux. I am using more command to view the contents of a file. If the file has many pages i am using f to move forward to the next page. But when i press f it skips to two pages instead of one page. i checked the man more. It shows the default is 1. Please share your... (1 Reply)
Discussion started by: nokiak810
1 Replies

2. Programming

array type has incomplete element type

Dear colleagues, One of my friend have a problem with c code. While compiling a c program it displays a message like "array type has incomplete element type". Any body can provide a solution for it. Jaganadh.G (1 Reply)
Discussion started by: jaganadh
1 Replies

3. Shell Programming and Scripting

String type to date type

Can one string type variable changed into the date type variable. (1 Reply)
Discussion started by: rinku
1 Replies

4. UNIX for Dummies Questions & Answers

pages out

okay.. have monitored that the page out/h max and avg have been 0 for one week. is this possible...? huge applications will have swap space being utilised ... so i am suprised that for one whole week, it has actually been zero please advise thanks (1 Reply)
Discussion started by: yls177
1 Replies

5. UNIX for Dummies Questions & Answers

man pages

Hi folks, I want to know all the commands for which man pages are available. How do i get it? Cheers, Nisha (4 Replies)
Discussion started by: Nisha
4 Replies

6. UNIX for Advanced & Expert Users

inactive pages

hi, plz tell me, how can get the inactive pages in HP UX. bye.... (1 Reply)
Discussion started by: venkat_t
1 Replies

7. UNIX for Advanced & Expert Users

wired pages

hi, can any body tell, what are wired pages in HP_UX. which structure contains that and plz tell the corresponding system call to get it. (1 Reply)
Discussion started by: venkat_t
1 Replies

8. UNIX for Dummies Questions & Answers

man pages

Hi, I've written now a man pages, but I don't knwo how to get 'man' to view them. Where have I to put this files, which directories are allowed?? THX Bensky (3 Replies)
Discussion started by: bensky
3 Replies

9. UNIX for Dummies Questions & Answers

Man pages

Hello , I just installed openssh in my system . I actually tried to man sshd but it says no entry , though there is a man directory in the installation which have the man pages for sshd . Can anyone tell me how should i install these man pages . DP (2 Replies)
Discussion started by: DPAI
2 Replies
Login or Register to Ask a Question