Login or Register to Ask a Question and Join Our Community


Solaris

Oracle memory usage on Solaris box

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Solaris Oracle memory usage on Solaris box
# 1  
Old 09-28-2010
Oracle memory usage on Solaris box
I am working on Oracle 2 node RAC 10.2.0.4 on Solaris 10 T2000 kit.

The box has around 32G of memory of which 24G is used by oracle user. There is 3G of free memory on the box.
Sga max is set to 5G and while checking v$pgastat i see that maximum pga memory memory allocated was 6.5G. So oracle database is using around 12G of memory. There are no other processes running on the box except for the clusterware processes and the database.

Code:
NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU
  2958 oracle     30G   24G    75% 187:45:36 1.9%
    69 root      304M  420M   1.3%  19:54:52 0.1%
     1 daemon   4648K 8688K   0.0%   3:30:49 0.2%
     1 smmsp    1352K 8472K   0.0%   0:00:04 0.0%

There are around 2500 connections to this instance.
From the ps -ef OS command I see around 300 each of these connections.

Code:
/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyC
/oracle/product/cluster/jdk/jre/bin/sparcv9/java -classpath /or

When I check prstat for specific /oracle/product/cluster/jdk/jre/bin/sparcv9/java process I see that
Code:
PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 18696 oracle     77M   28M sleep   29   10   0:14:27 0.0% java/17

For /usr/bin/ssh process
Code:
PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  2756 oracle   6328K 3032K sleep   29   10   0:00:00 0.0% ssh/1

I am not sure about how the remaining 12G is used by oracle sessions?
How can I confirm the total memory usage by java process and the memory usage by clusterware?
Am I missing something here?
Last edited by DukeNuke2; 09-28-2010 at 02:23 PM..
# 2  
Old 09-28-2010
Some diagnostic tools (like prstat for example) are summing (incorrectly) the shared memory for every process attached to it.
So probably you have more free memory - use sar (-r) or vmstat (free) to check the memory usage.
# 3  
Old 09-29-2010
Code:
sar -r

SunOS sunfire130 5.10 Generic_141414-07 sun4v    09/29/2010

00:00:00 freemem freeswap
00:10:00  379711  6365702
00:20:01  381464  6392244
00:30:00  370198  6201557
00:40:00  381115  6378368
00:50:00  380526  6374958
01:00:00  382979  6412045
01:10:00  380202  6374879
01:20:00  378190  6326520
01:30:01  380685  6361740
01:40:00  380966  6370946
01:50:00  379275  6349503
02:00:00  370904  6199362
02:10:00  377037  6310516
02:20:01  376730  6299128
02:30:00  377665  6307954
02:40:00  375837  6282499
02:50:00  373060  6250497
03:00:00  384821  6404191
03:10:00  396092  6567784
03:20:01  376387  6298590
03:30:00  353518  5895149
03:40:00  362947  6072349
03:50:00  362284  6060109
04:00:00  362085  6047641
04:10:00  359961  6006132
04:20:01  359425  6006543
04:30:00  358139  5998141
04:40:00  355569  5954072
04:50:00  353991  5926019
05:00:00  343458  5734082
05:10:00  352976  5902494
05:20:01  351096  5873102
05:30:00  348735  5838498
05:40:00  348238  5843592
05:50:00  345854  5798027
06:00:00  345263  5774854
06:10:01  355511  5948726
06:20:00  462428  8290025
06:30:00  572195 10711797
06:40:00  479550  8643372
06:50:00  415216  7203634
07:00:00  370971  6214658
07:10:00  366112  6108073
07:20:01  363548  6062562
07:30:00  372455  6188077
07:40:00  383153  6338427
07:50:00  381317  6322173

Average   378082  6374289

pagesize
8192

 vmstat 5
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m3 m4   in   sy   cs us sy id
 0 0 0 5882040 5128568 494 1837 98294413907 157 155 0 0 0 3 -0 -0 16458 30336 15693 4 3 94
 0 2 0 2979128 2867880 563 2089 218 1866 1866 0 0 0 0 0 0 19741 69288 19390 5 4 92

So seems like there is around 3G free memory which does match with the vmstat o/p

I am concerned about this issue as we did face memory contingency on this node when the other node failed over resulting in around 5000 sessions using this node.
Sessions were failing constantly due to memory issues, the issue was sorted only after fixing the other node. There was huge amount of swapping going on too.

sga max is set to 5G which is 1/6th of total RAM size. So this would mean more than 25G is consumed by PGA when failover occurs but when I check the PGA usage on both the instances it is around 6G per instance, so the combined usage is less than 12G and thats the reason I am confused.

So I am not sure whether there is some other clusterware process or java process that is consuming all the memory. From the ps -ef OS command I see around 300 each of these connections.

Code:
/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyC
/oracle/product/cluster/jdk/jre/bin/sparcv9/java -classpath /or

So I am not sure how to check the total memory consumption of these processes as I cannot sum them as this might give me a false value due to child processes involved. Is there any way for me to identify the actual memory usage of these processes?
Last edited by radoulov; 09-29-2010 at 05:14 AM.. Reason: Added code tags.
# 4  
Old 09-29-2010
Quote:
[...]
Sessions were failing constantly due to memory issues
Could you provide more details - what are the exact error messages?

Quote:
There was huge amount of swapping going on too
Which command you use to check if swapping occurs?

Quote:
So this would mean more than 25G is consumed by PGA when failover occurs
[...]
You don't have enough evidence for now.

Quote:
when I check the PGA usage on both the instances it is around 6G per instance,
so the combined usage is less than 12G and thats the reason I am confused
And how do you check the PGA status?

Quote:
So I am not sure how to check the total memory consumption of these processes as I cannot sum them
as this might give me a false value due to child processes involved
Why do you care for child processes? What's the sum of the memory used by these processes?
# 5  
Old 09-29-2010
Quote:
Could you provide more details - what are the exact error messages?
We were receiving out of process memory errors - unable to allocate .. bytes of memory for PGA

Quote:
Which command you use to check if swapping occurs?
vmstat - sr was increasing constantly and used oracle EM to monitor memory usage

Quote:
So this would mean more than 25G is consumed by PGA when failover occurs
Yes I cant comment on this now as I dont have facts for this.

Quote:
And how do you check the PGA status?
I am an Oracle DBA with little knowledge on Unix. I can check within Oracle by checking v$pgastat , v$process etc

Quote:
Why do you care for child processes? What's the sum of the memory used by these processes?
As I said earlier I see 259 java processes each consuming 26M RSS which comes to 6.7G.
There are same number of ssh processes each consuming around 2.5M which comes to 650M.
Is there any solaris command that will directly give the sum of memory used by all these processes?
I am using something like this to calculate the total memory usage of these processes

Code:
ps -ef | grep /oracle/product/cluster/jdk/jre/bin/sparcv9/java | wc -l
258
prstat -p 19053,25131,18696
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 18696 oracle     77M   26M sleep   29   10   0:14:27 0.0% java/17
 25131 oracle     77M   26M sleep   29   10   0:14:19 0.0% java/17
 19053 oracle     77M   26M sleep   29   10   0:14:21 0.0% java/17
ps -ef | grep PasswordAuthentication=no | wc -l
259
prstat -p 2756, 8622, 2230
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  2756 oracle   6328K 2432K sleep   29   10   0:00:00 0.0% ssh/1
  8622 oracle   6328K 2504K sleep   29   10   0:00:00 0.0% ssh/1
  2230 oracle   6328K 2528K sleep   29   10   0:00:00 0.0% ssh/1

Based on the average RSS size I am multuiplying it by total processes to compute total memory usage of these processes, is this the right way of computing the memory usage of these processes?
# 6  
Old 09-29-2010
Quote:
We were receiving out of process memory errors - unable to allocate .. bytes of memory for PGA
Could you provide the exact error code and message (ORA- ...)?

Quote:
vmstat - sr was increasing constantly and used oracle EM to monitor memory usage
The scan rate is not enough, check if si/so have values different than 0
(or if pi/po have high values) with vmstat -S.

Quote:
I can check within Oracle by checking v$pgastat , v$process etc
Could you post the output from:

Code:
set pages 200 lines 132
col name for a40
select * from v$pgastat;

Quote:
Based on the average RSS size I am multuiplying it by total processes to compute total memory usage
of these processes, is this the right way of computing the memory usage of these processes?
Just sum the RSS values of all those processes:

Code:
ps -eorss,args |
  nawk 'END { 
    print s/1024, "MB"
    }
    /PasswordAuthentication=no/ {
      s += $1
      }'



---------- Post updated at 11:53 AM ---------- Previous update was at 11:42 AM ----------

Could you also post the output of the following command:

Code:
set pages 200 lines 132
select server, count(1) from v$session
group by server;

# 7  
Old 09-29-2010
Quote:
Could you provide the exact error code and message (ORA- ...)?
ORA-04030: out of process memory when trying to allocate 82456 bytes (pga heap,control file i/o buffer)

Code:
SQL> !vmstat -S
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  si  so pi po fr de sr m0 m1 m3 m4   in   sy   cs us sy id
 0 0 0 5864088 5114360 0  0 114122802779 166 164 0 0 1 3 -0 -0 16470 30371 15706 4 3 94

Number of sessions

Code:
SQL> set pages 200 lines 132
select server, count(1) from v$session
group by server;SQL>   2

SERVER      COUNT(1)
--------- ----------
DEDICATED       2538

PGA usage

Code:
NAME                                          VALUE UNIT
---------------------------------------- ---------- ------------
aggregate PGA target parameter           2097152000 bytes
aggregate PGA auto target                 131072000 bytes
global memory bound                       209715200 bytes
total PGA inuse                          3829838848 bytes
total PGA allocated                      6324478976 bytes
maximum PGA allocated                    6921444352 bytes
total freeable PGA memory                 641400832 bytes
process count                                  2561
max processes count                            2850
PGA memory freed back to OS              7.7713E+10 bytes
total PGA used for auto workareas           2102272 bytes
maximum PGA used for auto workareas       150141952 bytes
total PGA used for manual workareas               0 bytes
maximum PGA used for manual workareas        537600 bytes
over allocation count                        400469
bytes processed                          1.9377E+11 bytes
extra bytes read/written                 6706253824 bytes
cache hit percentage                          96.65 percent
recompute count (total)                      403688

19 rows selected.

Memory usage by java processes and ssh processes

Code:
ps -eorss,args | nawk 'END { print s/1024, "MB" } /ssh/ { s += $1 }'
658.383 MB
ps -eorss,args | nawk 'END { print s/1024, "MB" } /java/ { s += $1 }'
5800.34 MB

I did try checking the rest of the processes other than oracle db, asm, ssh, java and external connections
Code:
ps -eorss,args | grep -v oracleavt31 | grep -v java | grep -v PasswordAuthentication=no | grep -v oracleavt31 | grep -v asm | grep -v ora_ | nawk 'END { print s/1024, "MB" }  { s += $1 }'
1281.53 MB

So Oracle SGA+PGA comes to around 11G, ASM sga settings is 300M, ssh memory usage is 658M, java memory usage is 5.8G, other process memory is around 1.3G. So in total it comes to around 19G but still I am missing 5G of memory, not sure how this is being utilized?

prstat -a -s rss shows oracle user being using 24G of memory.

Also for a few processes the complete command is getting trimmed for example
Code:
ps -ef | grep PasswordAuthentication
oracle  2230  2190   0   Sep 16 ?           0:00 /usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o StrictHostKeyC

and this is the same case when I check java processes too
Is there any way I can get to know the complete OS command being used without getting trimmed?
Last edited by prabumohan; 09-29-2010 at 07:49 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Solaris

How to find out the memory usage in Solaris?

Hi All, In one of the solaris box aslert got triggered as ... (Used_Real_Mem_Pct=93.0 Used_Swap_Space_Pct=75.0 )] when i see the usage by vmstat and sar i am not able to relate the alert with the free memory and swap memory please help to understand the vmstat output as below.. kthr ... (4 Replies)
Discussion started by: Riverstone
4 Replies

2. Solaris

Solaris memory usage

One of my Solaris Unix server has total RAM 128G . top & vmstat command shows free memory is 86G and usage is 42G. ps -eo pid,rss,commshows memory usage by process. when i sum all the memory usage by the process, it shows 1.9TB as shown below. $ ps -eo pid,pmem,vsz,rss,comm | sort -rnk2 |... (5 Replies)
Discussion started by: baladelaware73
5 Replies

3. Solaris

Solaris Memory usage by oracle, java

Hello Gurus, In Office, I have unix Solaris Server. Oracle and Java is installed on Solaris Server. Oracle when starts then oracle loads SGA, PGA memory and graps solaris server memory. In the same manner java and other applications grabs memory for them. I want memory distribution of... (4 Replies)
Discussion started by: get_chandrakant
4 Replies

4. Solaris

Memory usage in a Solaris Container

Hi All, We have a server with Solaris 10 installed. The total memory of the server is 64GB. In order to check the memory info of the server I use "top" utility that gives me total and free memory in real-time. I have also installed a Sun container (non-global zone) on top of parent operating... (4 Replies)
Discussion started by: Slayer
4 Replies

5. Solaris

Memory usage in Solaris - memory not freed?

Hi, I'm running a multi-process software system on a Solaris 8 machine. When I monitor the memory usage, I see that the free memory is dropping rapidly, but I can't detect a process that uses this memory. I'm using "top" to get the free memory and the memory usage of processes. Thanks. (3 Replies)
Discussion started by: gewurtz
3 Replies

6. HP-UX

how can I find cpu usage memory usage swap usage and logical volume usage

how can I find cpu usage memory usage swap usage and I want to know CPU usage above X% and contiue Y times and memory usage above X % and contiue Y times my final destination is monitor process logical volume usage above X % and number of Logical voluage above can I not to... (3 Replies)
Discussion started by: alert0919
3 Replies

7. UNIX for Dummies Questions & Answers

memory usage : kernel, oracle, other users

Hello, I'm looking for a command or script that could permit me to know what is memory usage group by HP-UX kernel, oracle processes and other users (specific software). A command like 'top' but for all processes and not only for 12 max could be very useful Thanks in advance (4 Replies)
Discussion started by: thierryUX
4 Replies

8. UNIX for Dummies Questions & Answers

Comman/Tool to check memory leak in solaris box

Hello Guys, Please let me know about commands or tools that are available to check memory leak in solaris box. Thanks in Advance (1 Reply)
Discussion started by: rajeshlbryan
1 Replies

9. Solaris

oracle process and LWP on solaris 8 box

Hi all, Without oracle tunning I understand that oracle processes always run on a single CPU at a time (the prstat output always show 1 CPU number) even on multi-processor hardware. Is it correct? Regards, (1 Reply)
Discussion started by: solea
1 Replies

10. Filesystems, Disks and Memory

Memory usage in the box

Hello: Environment is: Oracle 817 on IBM RS/6000 AIX 433 I have 4GB RAM on the box and Page/Swap is about the same. Presently I am using close to 1GB of RAM towards 5 instances of ORACLE production environments. How can I know, how much of memory/RAM is used for : Oracle Processes , I... (2 Replies)
Discussion started by: ST2000
2 Replies
Login or Register to Ask a Question