Where memory is being used?


 
Thread Tools Search this Thread
Operating Systems AIX Where memory is being used?
# 1  
Old 09-19-2011
Where memory is being used?

we have a database server with not much load and two good processors (4 core) on AIX 6.1. The Oracle becomes slow some time, but not all the time. I suspect the server is memory constraints. In order to justify the memory upgrade I am trying to see how much memory is being used by each process. When I look at following OS command results I am wondering where all the memory goes.
ps aux command gives account for only 46% of memory usage. Topas says 94% memory usage and vmstat says no paging. I am not an SA, what should I conclude from following.
Thanks,
Code:
/home/oracle> date;ps aux | head -1 ; ps aux | sort -rn +3 | head -30
Mon Sep 19 12:28:00 EDT 2011
USER          PID %CPU %MEM   SZ  RSS    TTY STAT    STIME  TIME COMMAND
appuser   9175156  0.0  6.0 233632 115176      - A      Sep 08 17:27 /was/wasadm/IB
oracle    8978536  0.0  2.0 96828 32148      - A      Sep 08  0:06 oracleprod (L
oracle    8650924  0.0  2.0 97200 43652      - A    12:27:06  0:00 ora_j000_prod
oracle    8323326  0.0  2.0 96368 31768      - A      Sep 07  0:00 ora_q001_prod
oracle    8257788  0.0  2.0 96440 31400      - A      Sep 07  0:00 ora_q000_prod
oracle    7340274  0.0  2.0 96304 31776      - A      Sep 07  0:01 ora_qmnc_prod
oracle    6881410  0.0  2.0 106764 32148      - A      Sep 07  0:04 ora_arc0_prod
oracle    6815974  0.0  2.0 96960 32764      - A      Sep 07  0:58 ora_ckpt_prod
oracle    6684684  0.0  2.0 96276 32500      - A      Sep 07  0:27 ora_mman_prod
oracle    6619376  0.0  2.0 96804 32624      - A      Sep 07  0:15 ora_lgwr_prod
oracle    6488266  0.0  2.0 96480 32044      - A      Sep 07  0:00 ora_arc1_prod
oracle    6422742  0.0  2.0 100988 32204      - A      Sep 07  0:00 ora_s000_prod
oracle    6291658  0.0  2.0 96492 32904      - A      Sep 07  1:23 ora_mmnl_prod
oracle    6160574  0.0  2.0 96568 32120      - A      Sep 07  0:00 ora_reco_prod
oracle    6094870  0.0  2.0 97220 31832      - A      Sep 07  0:00 ora_d000_prod
oracle    5963852  0.0  2.0 101252 32680      - A      Sep 07  0:44 ora_pmon_prod
oracle    5898488  0.0  2.0 96272 31876      - A      Sep 07  0:21 ora_psp0_prod
oracle    5767408  0.0  2.0 99228 34800      - A      Sep 07  0:48 ora_mmon_prod
oracle    5439674  0.0  2.0 98292 35468      - A      Sep 07  1:58 ora_cjq0_prod
oracle    4587728  0.0  2.0 98236 34092      - A      Sep 07  0:18 ora_smon_prod
oracle    3735706  0.0  2.0 99268 32604      - A      Sep 07  0:20 ora_dbw0_prod
root      4718626  0.0  1.0 83104 26748      - A      Sep 07  2:34 /var/opt/tivol
root      9961508  0.0  0.0  816  828      - A    12:20:59  0:00 telnetd -a
root      9306142  0.0  0.0 3988 1408      - A      Sep 07  0:07 /usr/sbin/rsct
root      8388612  0.0  0.0 1660   20      - A      Sep 07  0:00 /usr/sbin/rsct
root      7995638  0.0  0.0  616  124      - A      Sep 07  2:40 /usr/sbin/gett
root      7602412  0.0  0.0 4068  820      - A      Sep 07  0:26 /usr/bin/topas
root      7471104  0.0  0.0 1812   20      - A      Sep 07  0:00 /usr/sbin/rsct
root      7405576  0.0  0.0 10168  496      - A      Sep 07  0:14 /usr/sbin/rsct
root      6750330  0.0  0.0 3168  204      - A      Sep 07  0:02 bin/nonstop_ai

Following from TOPAS
Code:
MEMORY
Real,MB    2048
% Comp     94
% Noncomp   2
% Client    2

Following from VMSTAT
Code:
/home/oracle> vmstat 3 5

System configuration: lcpu=8 mem=2048MB ent=1.50

kthr    memory              page              faults              cpu
----- ----------- ------------------------ ------------ -----------------------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
 0  0 760997 12289   0  17   0   0    0   0  30 1703 530  1  1 97  1  0.05   3.1
 0  0 760997 12286   0   1   0   0    0   0   4  214 465  0  0 99  0  0.02   1.1
 0  0 760997 12286   0   0   0   0    0   0   0   71 187  0  1 99  0  0.02   1.2
 0  0 760997 12286   0   0   0   0    0   0   2  179 456  0  0 99  0  0.02   1.0
 0  0 760997 12286   0   0   0   0    0   0   2  199 452  0  0 99  0  0.01   0.9



Last edited by Scott; 09-19-2011 at 12:56 PM.. Reason: Please use code tags
# 2  
Old 09-19-2011
Check
Code:
ipcs -m

This User Gave Thanks to bartus11 For This Post:
# 3  
Old 09-19-2011
following is the output of ipcs. Topas still shows 95% comp memory usage.


Code:
/home/oracle> ipcs -mb
IPC status from /dev/mem as of Mon Sep 19 12:59:55 EDT 2011
T        ID     KEY        MODE       OWNER    GROUP     SEGSZ
Shared Memory:
m   1048576 0x7800002c --rw-rw-rw-     root   system 268435456
m   2097153 0x78000020 --rw-rw-rw-     root   system  16777216
m   1048578 0x4f1f16e0 --rw-r-----   oracle      dba 612376576
m         3 0x21002007 --rw------- pconsole   system  10485760
m         4 0x0d0009a0 --rw-rw----     root   system      1440
m         5 0x2100200d --rw-rw----     root   system  52428800


Last edited by Scott; 09-19-2011 at 05:01 PM.. Reason: Added code tags
# 4  
Old 09-19-2011
Despite the fact that your data is very apparently taken while nothing was happening on the DB rather than when you were facing your issues, I can still tell you that you actually use 1 GB memory computational that you do not have - do lsps -a and you will see a significant amount of paging space used - most likely AIX has moved parts of your DB entirely to disk - BAD THING

your box is hopelessly overutilized. How much memory oracle is really using is something you can see with svmon -U oracle

Please note as well that the system needs memory for filesystem buffering, for any kind of connection, for each oracle process .... this adds up - and the memory you have assigned to your SGA / PGA is not available in first instance.

Regards
zxmaus
# 5  
Old 09-19-2011
Thanks XZMAUS for your reply.
lsps looks bad. Is it 40% of both disks used for paging? vmstat do not show anything under pi and po.

/home/oracle> lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum
hd6 hdisk0 rootvg 528MB 40 yes yes lv 0
hd6 hdisk3 rootvg 3184MB 40 yes yes lv 0

here is the trimmed output of svmon -U oracle. How to find what is paged out from following list?

Code:
===============================================================================
User                                 Inuse      Pin     Pgsp  Virtual
oracle                               90121    10167   162819   224644

     PageSize                Inuse        Pin       Pgsp    Virtual
     s    4 KB               59897         23     162819     194420
     m   64 KB                1889        634          0       1889

...............................................................................
SYSTEM segments                      Inuse      Pin     Pgsp  Virtual
                                      9050     8195     2603    11536

     PageSize                Inuse        Pin       Pgsp    Virtual
     s    4 KB                 154          3       2603       2640
     m   64 KB                 556        512          0        556

    Vsid      Esid Type Description              PSize  Inuse   Pin Pgsp Virtual
    4002         0 work kernel segment               m    556   512    0     556
    a005  9ffffffd work shared library              sm     98     0 2218    2218


...............................................................................
EXCLUSIVE segments                   Inuse      Pin     Pgsp  Virtual
                                     26662     1972    54931    76108

     PageSize                Inuse        Pin       Pgsp    Virtual
     s    4 KB               23334         20      54931      72780
     m   64 KB                 208        122          0        208

    Vsid      Esid Type Description              PSize  Inuse   Pin Pgsp Virtual
  8024a1        10 clnt text data BSS heap,          s   7631     0    -       -
                        /dev/orabinlv:117014                                   
  821350        11 work text data BSS heap          sm   2274     0    0    2274
  8369bb         - clnt /dev/oradblv:27              s   1311     0    -       -
  82b0f5  fffffff0 work application stack           sm      0     0    0       0

...............................................................................
SHARED segments                      Inuse      Pin     Pgsp  Virtual
                                     54409        0   105285   137000

     PageSize                Inuse        Pin       Pgsp    Virtual
     s    4 KB               36409          0     105285     119000
     m   64 KB                1125          0          0       1125

    Vsid      Esid Type Description              PSize  Inuse   Pin Pgsp Virtual
  8065a3  70000002 work default shmat/mmap          sm  22676     0 53603   58256
    9000         d work shared library text          m    817     0    0     817
  80e5e7  70000001 work default shmat/mmap          sm   6746     0 38262   43824
  8165eb  70000003 work default shmat/mmap          sm   6629     0 13138   16630
  82a015  90000000 work shared library text          m    308     0    0     308
  83c8fe         1 clnt code,/dev/hd2:3048           s    150     0    -       -
  82e017  90020014 work shared library               s    125     0  282     290
  812189         1 clnt code,/dev/hd2:151            s     66     0    -       -
  8320b9  9fffffff clnt USLA text,/dev/hd2:4218      s     13     0    -       -
  830138         - clnt /dev/hd2:25928               s      2     0    -       -
  800d40         - clnt /dev/hd2:26059               s      2     0    -       -


Last edited by AIX_DBA; 09-19-2011 at 05:16 PM.. Reason: Code tags, please...
# 6  
Old 09-20-2011
Code:
User                                 Inuse      Pin     Pgsp  Virtual
oracle                               90121    10167   162819   224644

the number under pgsp is what is actually in paging space and owned by oracle - close to 2/3 of your entire oracle memory utilization - no wonder that your DB is slow

if AIX has moved enough into paging space to have sufficient memory to operate than you will not see any 'new' paging - page ins / page outs - until you start a very work intensive process. The system will at that moment probably start extensive scan and free until it gives up and just moves more content into paging.

From the data you have posted so far, the system is running apart from oracle websphere too - as far as I remember you assign memory to websphere too similar to SGA + PGA to oracle - this is physical memory not available any more to the system.

As your oracle memory utilization seems decent, I would assume that websphere is consuming a large chunk of memory for itself.

Can you maybe post the output of svmon -U appuser too ?

Regards
zxmaus
# 7  
Old 09-23-2011
zamaus thank you for responding to my queries. Some how I never got notification about this your last update to the thread, so I am responding late.

Code:
/home/oracle> date;ps aux | head -1 ; ps aux | sort -rn +3 | head -30
Fri Sep 23 17:32:51 EDT 2011
USER          PID %CPU %MEM   SZ  RSS    TTY STAT    STIME  TIME COMMAND
appuser  9175156  0.0  5.0 233680 110628      - A      Sep 08 23:31 /was/wasadm/IB
oracle   10289318  0.0  2.0 97200 45496      - A    17:32:14  0:00 ora_j000_bldml
oracle    8978536  0.0  2.0 96828 33980      - A      Sep 08  0:10 oraclebldml (L
oracle    8323326  0.0  2.0 96368 33612      - A      Sep 07  0:01 ora_q001_bldml
oracle    8257788  0.0  2.0 96440 33244      - A      Sep 07  0:00 ora_q000_bldml
oracle    7340274  0.0  2.0 96304 33620      - A      Sep 07  0:02 ora_qmnc_bldml
oracle    6881410  0.0  2.0 106764 33996      - A      Sep 07  0:07 ora_arc0_bldml
oracle    6815974  0.0  2.0 97108 34752      - A      Sep 07  1:19 ora_ckpt_bldml
oracle    6684684  0.0  2.0 96276 34344      - A      Sep 07  0:37 ora_mman_bldml
oracle    6619376  0.0  2.0 96804 34460      - A      Sep 07  0:21 ora_lgwr_bldml
oracle    6488266  0.0  2.0 106764 33940      - A      Sep 07  0:00 ora_arc1_bldml
oracle    6422742  0.0  2.0 100988 34048      - A      Sep 07  0:00 ora_s000_bldml
oracle    6291658  0.0  2.0 96492 34760      - A      Sep 07  1:54 ora_mmnl_bldml
oracle    6160574  0.0  2.0 96568 33964      - A      Sep 07  0:00 ora_reco_bldml
oracle    6094870  0.0  2.0 97220 33676      - A      Sep 07  0:00 ora_d000_bldml
oracle    5963852  0.0  2.0 101252 34524      - A      Sep 07  1:00 ora_pmon_bldml
oracle    5898488  0.0  2.0 96272 33720      - A      Sep 07  0:29 ora_psp0_bldml
oracle    5767408  0.0  2.0 99272 36804      - A      Sep 07  1:06 ora_mmon_bldml
oracle    5439674  0.0  2.0 98616 37176      - A      Sep 07  2:41 ora_cjq0_bldml
oracle    4587728  0.0  2.0 98236 36044      - A      Sep 07  0:24 ora_smon_bldml
oracle    3735706  0.0  2.0 99268 34452      - A      Sep 07  0:28 ora_dbw0_bldml
root      4718626  0.0  1.0 83168 24332      - A      Sep 07  3:26 /var/opt/tivol
root      9306142  0.0  0.0 4196 1400      - A      Sep 07  0:10 /usr/sbin/rsct
root      8388612  0.0  0.0 1660   20      - A      Sep 07  0:00 /usr/sbin/rsct
root      7995638  0.0  0.0  616  124      - A      Sep 07  3:44 /usr/sbin/gett
root      7602412  0.0  0.0 4068  800      - A      Sep 07  0:36 /usr/bin/topas
root      7471104  0.0  0.0 1812   20      - A      Sep 07  0:00 /usr/sbin/rsct
root      7405576  0.0  0.0 10168  496      - A      Sep 07  0:19 /usr/sbin/rsct
root      6750330  0.0  0.0 3168  204      - A      Sep 07  0:04 bin/nonstop_ai
root      6029498  0.0  0.0 28440 1332      - A      Sep 07  0:03 [cimserve]

/home/oracle> svmon -U  appuser
===============================================================================
User                                 Inuse      Pin     Pgsp  Virtual
appuser                              *** user does not exist ***


Last edited by Scott; 09-23-2011 at 05:40 PM.. Reason: Code tags
This User Gave Thanks to AIX_DBA For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Solaris

[DOUBT] Memory high in idle process on Solaris 10 (Memory Utilization > 90%)

Hi Experts, Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing. Why memory utilization on solaris always looks high? I have statement about memory on solaris, is this true: Memory in solaris is used for... (4 Replies)
Discussion started by: edydsuranta
4 Replies

2. Solaris

relationship or difference between entitled memory and locked memory

Hello solaris experts, Being new to solaris containers, from Linux, feeling difficulty in understanding certain concepts. Hope somebody can help me here. I understand that, & some questions .... Locked memory -- memory which will not be swapped out at any cause. is this for... (0 Replies)
Discussion started by: thegeek
0 Replies

3. UNIX for Advanced & Expert Users

Out of Memory error when free memory size is large

I was running a program and it stopped and showed "Out of Memory!". at that time, the RAM used by this process is around 4G and the free memory size of the machine is around 30G. Does anybody know what maybe the reason? this program is written with Perl. the OS of the machine is Solaris U8. And I... (1 Reply)
Discussion started by: lilili07
1 Replies

4. Solaris

restrcit physical memory with zone.max-locked-memory

Is it possible to restrict physical memory in solaris zone with zone.max-locked-memory just like we can do with rcapd ? I do not want to used rcapd (1 Reply)
Discussion started by: fugitive
1 Replies

5. Programming

How to deal with lots of data in memory in order not to run out of memory

Hi, I'm trying to learn how to manage memory when I have to deal with lots of data. Basically I'm indexing a huge file (5GB, but it can be bigger), by creating tables that holds offset <-> startOfSomeData information. Currently I'm mapping the whole file at once (yep!) but of course the... (1 Reply)
Discussion started by: emitrax
1 Replies

6. Solaris

How to find Total and Free Physical Memory and Logical Memory in SOLARIS 9

Hi, Im working on Solaris 9 on SPARC-32 bit running on an Ultra-80, and I have to find out the following:- 1. Total Physical Memory in the system(total RAM). 2. Available Physical Memory(i.e. RAM Usage) 3. Total (Logical) Memory in the system 4. Available (Logical) Memory. I know... (4 Replies)
Discussion started by: 0ktalmagik
4 Replies

7. AIX

ulimits max locked memory virtual memory

Hi, Would any one be so kind to explain me : are ulimits defined for each user seperately ? When ? Specialy what is the impact of : max locked memory and virtual memory on performance of applications for a user. Many thanks. PS : this is what I can see in MAN : ulimit ] ... (5 Replies)
Discussion started by: big123456
5 Replies

8. HP-UX

How can I get memory usage or anything that show memory used from sar file?

Refer from title: How can i get memory used or anything that can show memory from sar file example on solaris:- we can use sar with option to show memory used at time that sar crontab run. on HP-UX, it not has option to see memory used. But i think it may be have some parameter or some... (1 Reply)
Discussion started by: panithat
1 Replies

9. Programming

how to round up a memory address(memory alignment problem)

Hi, I try to marshal a unsigned int and a char * into a buffer, and then unmarshal them later to get them out. I need to put the char * in the front and unsigned int at the end of the buffer. However, my system always give me "BUS ERROR". I am using Sun Sparcs Sloris 2.10. My code to marshal... (6 Replies)
Discussion started by: nj302
6 Replies
Login or Register to Ask a Question