Oracle memory usage on Solaris box

09-29-2010

Registered User

5,690, 630

Join Date: Jan 2007

Last Activity: 9 January 2017, 4:40 AM EST

Location: Варна, България / Milano, Italia

Posts: 5,690

Thanks Given: 184

Thanked 630 Times in 587 Posts

Quote:

ORA-04030: out of process memory when trying to allocate 82456 bytes (pga heap,control file i/o buffer)

You need far more PGA(>=6) than your current target(2G).
Bump it up to at least 8G.

Quote:

!vmstat -S

Without options, vmstat displays a one-line summary
of the virtual memory activity since the system was booted.
You need at least vmstat -S 5 50 to get useful result.

Quote:

ps -eorss,args | nawk 'END { print s/1024, "MB" } /java/ { s += $1 }'
5800.34 MB

Try to understand what those java processes are doing,
you can display more details with /usr/ucb/ps auxwww | fgrep java.

Quote:

prstat -a -s rss shows oracle user being using 24G of memory.

I already explained why prstat summary output should not be considered!

radoulov

View Public Profile for radoulov

Find all posts by radoulov

09-29-2010

Registered User

14, 0

Join Date: Sep 2010

Last Activity: 8 December 2010, 4:57 AM EST

Posts: 14

Thanks Given: 0

Thanked 0 Times in 0 Posts

Thank you very much.

This pga_aggregate_target parameter which is set is just a target value isn't it, so even if it is small and if Oracle needs more memory it would go for it, is that not the case?
Do you still recommend setting PGA to around 8G, is this just to make sure Oracle reserves that memory so that no other process can make use of it when Oracle actually needs it?

Regarding Oracle processes, these are all the processes using SGA memory but I see different values for each process, so should I conclude they all will be using memory within the SGA space or will they be stretching outside the SGA memory?

Code:

2686896 ora_psp0_avt32
2715088 ora_lms1_avt32
3261872 ora_j002_avt32
2715336 ora_lms2_avt32
3260336 ora_pz99_avt32
3227800 ora_asmb_avt32
2714480 ora_pmon_avt32
3219688 ora_q000_avt32
2710496 ora_dbw3_avt32
2725976 ora_smon_avt32
2733528 ora_s000_avt32
2953304 ora_dmon_avt32
3235696 ora_arc1_avt32
2715216 ora_lms5_avt32
3235688 ora_arc0_avt32
2715256 ora_lms6_avt32
2715320 ora_lms4_avt32
2703896 ora_ckpt_avt32
2715096 ora_dbw0_avt32
2715248 ora_lms7_avt32
2715360 ora_lms0_avt32
3260368 ora_q001_avt32
2735736 ora_cjq0_avt32
2732736 ora_mmon_avt32
2710848 ora_lmon_avt32
2721312 ora_reco_avt32
2712400 ora_lck0_avt32
3211512 ora_qmnc_avt32
2705352 ora_lgwr_avt32
2622864 ora_dism_avt32
2722368 ora_mmnl_avt32
2692112 ora_d000_avt32
2707472 ora_mman_avt32
2710488 ora_dbw2_avt32
2715776 ora_lms3_avt32
2718952 ora_dbw1_avt32
2711952 ora_lmd0_avt32
3259624 ora_j004_avt32
4570520 ora_j005_avt32
3260064 ora_j003_avt32
3207096 ora_rbal_avt32
3262000 ora_j000_avt32
3206960 ora_insv_avt32
4571280 ora_j001_avt32
4570184 ora_j007_avt32
4579240 ora_pz98_avt32
4570712 ora_j006_avt32
4570632 ora_j008_avt32
4570824 ora_j009_avt32
4713888 ora_diag_avt32
4577648 ora_pz93_avt32
4576624 ora_pz92_avt32
4577560 ora_pz97_avt32
4577576 ora_pz94_avt32
4571656 ora_o002_avt32
4575800 ora_pz90_avt32
4577520 ora_pz95_avt32
4575800 ora_pz89_avt32
4584536 ora_emn0_avt32
4575800 ora_pz88_avt32
4577560 ora_pz96_avt32
4575928 ora_pz91_avt32

vmstat

Code:

vmstat -S 5 50
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  si  so pi po fr de sr m0 m1 m3 m4   in   sy   cs us sy id
 0 0 0 5855176 5107136 0  0 116114603935 168 166 0 0 1 3 -0 -0 16483 30378 15720 4 3 94
 0 0 0 2608536 2540264 0  0 124 69 65 0  0  0  0  0  0 17361 58626 16488 10 3 87
 0 0 0 2614104 2544112 0  0 13 22 22  0  0  0  0  0  0 16116 16941 15440 9 2 88
 0 0 0 2611808 2540504 0  0 1992 27 27 0 0  0  0  0  0 17254 18751 16453 9 2 89
 0 0 0 2609656 2538664 0  0 6808543396 27 27 0 0 1 0 0 0 16591 34676 15744 12 5 83
 0 0 0 2607216 2535224 0  0 550 19 19 0  0  0  0  0  0 18155 54435 17277 13 7 80
 0 0 0 2615648 2543304 0  0  0 19 17  0  0 11  0  0  0 16155 14767 15394 8 2 91
 0 0 0 2604184 2535800 0  0  5 79 79  0  0  0  0  0  0 17573 69001 16663 11 4 85
 0 0 0 2611160 2538176 0  0 6812339727 44 44 0 0 0 0 0 0 16395 17328 15547 11 2 87
 0 1 0 2614408 2542520 0  0 68131839621 16 16 0 0 0 0 0 0 17002 24987 16228 9 3 89
 0 0 0 2615720 2544024 0  0 6813183962 10 10 0 0 0 0 0 0 16178 15207 15384 9 2 89
 0 0 0 2615568 2544320 0  0  0 32 32  0  0  0  0  0  0 17183 17673 16274 9 2 88
 0 0 0 2616136 2544240 0  0  0 19 19  0  0 12  0  0  0 16114 14547 15347 8 2 91
 0 0 0 2615824 2544600 0  0  0 52 48  0  0  0  0  0  0 16909 43463 16170 9 2 88
 0 0 0 2609704 2538928 0  0  0 51 51  0  0  0  0  0  0 16059 33665 15071 10 3 87
 0 0 0 2615872 2543832 0  0  0 32 32  0  0  0  0  0  0 16753 17561 15976 8 2 90
 0 0 0 2614328 2542432 0  0 20437019104 17 17 0 0 0 0 0 0 17384 51736 16964 11 5 84
 0 0 0 2614536 2542352 0  0 6800541983 8 8 0 0 0 0 0 0 17110 22122 16306 10 3 87
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  si  so pi po fr de sr m0 m1 m3 m4   in   sy   cs us sy id
 0 0 0 2616296 2543568 0  0  0 13 13  0  0 10  0  0  0 16133 16474 15379 8 2 90
 0 0 0 2611760 2536872 0  0 13628056864 49 46 0 0 0 0 0 0 17087 30483 16141 10 5 85
 0 0 0 2611976 2536688 0  0  0 41 41  0  0  0  0  0  0 16444 57780 15615 9 3 88
 0 0 0 2615752 2539648 0  0  0 16 16  0  0  0  0  0  0 17011 25215 16246 9 3 88
 0 0 0 2616608 2540632 0  0  0  5  5  0  0  0  0  0  0 16541 15918 15687 9 2 89
 0 0 0 2616296 2540560 0  0  0 10  8  0  0  0  0  0  0 16883 17348 16141 9 2 89
 0 0 0 2616760 2541712 0  0  0  5  5  0  0  0  0  0  0 15824 14300 15109 8 2 91
 0 0 0 2616808 2543384 0  0 68123397012 24 22 0 0 0 0 0 0 16746 16781 15902 9 2 89
 0 0 0 2611800 2538968 0  0 165 44 44 0  0  0  0  0  0 16656 99192 15917 10 3 87
 0 0 0 2616400 2542888 0  0 6811073702 41 41 0 0 0 0 0 0 16702 19093 15879 8 2 89
 0 0 0 2615568 2541728 0  0 10  6  6  0  0  0  0  0  0 17296 53964 16710 12 6 82
 0 0 0 2616576 2541736 0  0 40876570832 8 8 0 0 0 0 0 0 16948 20027 16190 10 2 88
 0 0 0 2616000 2541648 0  0  0 13 13  0  0  0  0  0  0 15940 14917 15126 8 2 91
 0 0 0 2613592 2539904 0  0 457 11 11 0  0  1  0  0  0 17123 22652 16033 11 2 87
 0 0 0 2610776 2536480 0  0  0 47 47  0  0  0  0  0  0 16702 64147 15674 10 3 86
 0 0 0 2611488 2536424 0  0  2 38 38  0  0  0  0  0  0 17120 35360 15958 10 4 86
 0 0 0 2616784 2540584 0  0  0  0  0  0  0  0  0  0  0 16137 15592 15216 10 2 88
 0 0 0 2617640 2541408 0  0  0 32 32  0  0  0  0  0  0 17171 18370 16215 9 2 89
 0 0 0 2617568 2541704 0  0 6804750760 10 10 0 0 20 0 0 0 16780 18186 15833 8 2 90
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  si  so pi po fr de sr m0 m1 m3 m4   in   sy   cs us sy id
 0 0 0 2617584 2541880 0  0  0 17 16  0  0  0  0  0  0 16603 16552 15689 10 2 88
 0 0 0 2617336 2541912 0  0  0 29 29  0  0  0  0  0  0 16217 40843 15438 9 2 89
 0 0 0 2575912 2510144 0  0 166 55 55 0  0  0  0  0  0 16900 74873 16043 9 3 88
 0 0 0 2593848 2522744 0  0  0  6  6  0  0  0  0  0  0 17332 54147 16659 13 6 81
 0 0 0 2617928 2541632 0  0  0  5  5  0  0  0  0  0  0 16665 19349 15901 9 2 89
 0 0 0 2617704 2542160 0  0  0  3  3  0  0  9  0  0  0 16119 15255 15259 8 2 91
 0 0 0 2617744 2542272 0  0  0 14 14  0  0  0  0  0  0 16584 17990 15685 10 2 88
 0 1 0 2618048 2542344 0  0 13622147405 11 11 0 0 0 0 0 0 16125 15588 15364 8 2 90
 0 1 0 2601264 2538312 0  0  2 29 29  0  0  0  0  0  0 16964 65974 16176 9 3 88
 0 0 0 2616584 2541088 0  0  0 10 10  0  0  0  0  0  0 16269 15705 15327 10 2 88
 0 2 0 2617464 2541712 0  0 6811073702 29 29 0 0 0 0 0 0 17082 17984 16231 8 2 90
 0 1 0 2607328 2537256 0  0 105 3  3  0  0  0  0  0  0 19861 30992 19624 10 3 87
 0 2 0 2591376 2525472 0  0  0 11 11  0  0  0  0  0  0 17124 20198 16274 8 2 90

Its all similar java processes

Code:

   229  oracle   25479  0.0  0.17883226960 ?        S   Sep 16 15:14 /oracle/product/cluster/jdk/jre/bin/sparcv9/java -classpath /oracle/product/cluster/jdk/jre//lib/rt.jar:/oracle/product/cluster/jlib/cvu.jar:/oracle/product/cluster/jlib/srvm.jar -DCV_DESTLOC=/tmp -DCV_HOME=/oracle/product/cluster oracle.ops.verification.client.CluvfyDriver comp crs -display_status -n sunfire129,sunfire131,sunfire130

This is a Oracle clusterware process but not sure what this exact process is. Any suggestions?
I am also not sure why there are around 260 of these processes?

Last edited by prabumohan; 09-29-2010 at 08:27 AM..

prabumohan

View Public Profile for prabumohan

Find all posts by prabumohan

09-29-2010

Registered User

5,690, 630

Join Date: Jan 2007

Last Activity: 9 January 2017, 4:40 AM EST

Location: Варна, България / Milano, Italia

Posts: 5,690

Thanks Given: 184

Thanked 630 Times in 587 Posts

Quote:

This pga_aggregate_target parameter which is set is just a target value isn't it, so even if it is small and
if Oracle needs more memory it would go for it, is that not the case?

Yes, it's true. But it's also true that it will be less efficient
when it doesn't succeed to acquire the needed memory in one pass.

Quote:

Do you still recommend setting PGA to around 8G, is this just to make sure Oracle reserves that memory
so that no other process can make use of it when Oracle actually needs it?

No, I won't, given the output of vmstat ...
No, it's not about reserved memory, it's about more efficient overall memory management.

Quote:

Regarding Oracle processes, these are all the processes using SGA memory but I see different values
for each process, so should I conclude they all will be using memory within the SGA space
or will they be stretching outside the SGA memory?

They surely go beyond the sga memory - an oracle process attaches to a shared memory segment (the SGA)
and it also needs a private memory (its portion of the PGA). So what you see is normal.

It appears that you have a very busy system.
Your system need more memory that it currently has,
so you could:

1. Try to tune the Oracle processes in order to reduce memory usage.
2. Try to understand and tune the non oracle processes memory usage.

Consider that the Solaris kernel will try to optimize the memory usage distributing the available memory among file system cache and other areas, so it's quite difficult to get an exact image of the used memory.

I believe that your system is not properly sized to support the current load on a single node (i.e. to be able to handle the current load on a single node, you'll need more than 32 GB of physical memory).

radoulov

View Public Profile for radoulov

Find all posts by radoulov

09-29-2010

Registered User

14, 0

Join Date: Sep 2010

Last Activity: 8 December 2010, 4:57 AM EST

Posts: 14

Thanks Given: 0

Thanked 0 Times in 0 Posts

Thanks once again.

Quote:

It appears that you have a very busy system.
Your system need more memory that it currently has,

I am not sure why it should be so busy when SGA and PGA consumption are less than 11G at the moment. I do understand that PGA consumption would raise substantially during a failover but taking the present situation into account I have
11G of memory used by SGA+PGA
300M used by ASM
1.3G by other processes including root
6.5G used by cluster verify process(java) and ssh
3G free
which totals to 22.1G so still around 10G should be free, but not sure what is consuming that 10G?

I have identified one of the problems which is the cluster verify java process that has spawned 250 times and I dont see this process on any other cluster machines. This is only needed during installation so I am quite confused why is this one running now. I will dig around to see why this has happened.
The other problem is the missing 10G of memory and I don't have a clue how to get to the bottom of this one.
Any suggestions how I can check this out? Am I missing something here?

prabumohan

View Public Profile for prabumohan

Find all posts by prabumohan

09-29-2010

Registered User

5,690, 630

Join Date: Jan 2007

Last Activity: 9 January 2017, 4:40 AM EST

Location: Варна, България / Milano, Italia

Posts: 5,690

Thanks Given: 184

Thanked 630 Times in 587 Posts

Could you post the output of the following commands:

Code:

echo "::memstat" | mdb -k

Warning, the following command could be quite resource intensive:

Code:

kstat -m zfs

I'm just saying that the fact than you don't see that 10G as free memory, doesn't mean that it's not available (the kernel will allocate memory for file system caches and will release it as soon as some process needs it.

radoulov

View Public Profile for radoulov

Find all posts by radoulov

09-29-2010

Registered User

14, 0

Join Date: Sep 2010

Last Activity: 8 December 2010, 4:57 AM EST

Posts: 14

Thanks Given: 0

Thanked 0 Times in 0 Posts

Code:

echo "::memstat" | mdb -k
mdb: failed to open /dev/mem: Permission denied

I dont have permissions as I am not a sysadmin

Code:

 kstat -m zfs
module: zfs                             instance: 0
name:   arcstats                        class:    misc
        c                               32529580032
        c_max                           32529580032
        c_min                           4066197504
        crtime                          240.281628246
        deleted                         0
        demand_data_hits                0
        demand_data_misses              0
        demand_metadata_hits            0
        demand_metadata_misses          0
        evict_skip                      0
        hash_chain_max                  0
        hash_chains                     0
        hash_collisions                 0
        hash_elements                   0
        hash_elements_max               0
        hdr_size                        0
        hits                            0
        l2_abort_lowmem                 0
        l2_cksum_bad                    0
        l2_evict_lock_retry             0
        l2_evict_reading                0
        l2_feeds                        0
        l2_free_on_write                0
        l2_hdr_size                     0
        l2_hits                         0
        l2_io_error                     0
        l2_misses                       0
        l2_rw_clash                     0
        l2_size                         0
        l2_writes_done                  0
        l2_writes_error                 0
        l2_writes_hdr_miss              0
        l2_writes_sent                  0
        memory_throttle_count           0
        mfu_ghost_hits                  0
        mfu_hits                        0
        misses                          0
        mru_ghost_hits                  0
        mru_hits                        0
        mutex_miss                      0
        p                               16264790016
        prefetch_data_hits              0
        prefetch_data_misses            0
        prefetch_metadata_hits          0
        prefetch_metadata_misses        0
        recycle_miss                    0
        size                            0
        snaptime                        1244341.11405159

module: zfs                             instance: 0
name:   vdev_cache_stats                class:    misc
        crtime                          240.281860426
        delegations                     0
        hits                            0
        misses                          0
        snaptime                        1244341.1205129

Let me know if this of any help.

prabumohan

View Public Profile for prabumohan

Find all posts by prabumohan

09-29-2010

Registered User

5,690, 630

Join Date: Jan 2007

Last Activity: 9 January 2017, 4:40 AM EST

Location: Варна, България / Milano, Italia

Posts: 5,690

Thanks Given: 184

Thanked 630 Times in 587 Posts

Ask your sysadmin(s) to run the first command for you and send you the output.

This should give you an idea of the current memory usage of oracle user processes (assuming a single Oracle instance, some shared memory segments are missing):

Code:

ps -uoracle -opid= |
  xargs 2>/dev/null pmap -x |
    nawk 'END {
      for (p in priv) 
        printf "process: %s\n\t\tshared: %d KB\n\t\ttot private: %d KB\n", \
          p, shm[p], priv[p]        
      }
    /^[0-9]*:/ { 
      if (/ora/) {
        proc = "oracle"
        next
        }
      proc = $2 
      }
     /total/ {  
      $NF == "-" || shm[proc] = $NF
        priv[proc] += $5 
      }'

Last edited by radoulov; 09-29-2010 at 05:11 PM..

radoulov

View Public Profile for radoulov

Find all posts by radoulov

Solaris

Oracle memory usage on Solaris box

10 More Discussions You Might Find Interesting

1. Solaris

How to find out the memory usage in Solaris?

Discussion started by: Riverstone

2. Solaris

Solaris memory usage

Discussion started by: baladelaware73

3. Solaris

Solaris Memory usage by oracle, java

Discussion started by: get_chandrakant

4. Solaris

Memory usage in a Solaris Container

Discussion started by: Slayer

5. Solaris

Memory usage in Solaris - memory not freed?

Discussion started by: gewurtz

6. HP-UX

how can I find cpu usage memory usage swap usage and logical volume usage

Discussion started by: alert0919

7. UNIX for Dummies Questions & Answers

memory usage : kernel, oracle, other users

Discussion started by: thierryUX

8. UNIX for Dummies Questions & Answers

Comman/Tool to check memory leak in solaris box

Discussion started by: rajeshlbryan

9. Solaris

oracle process and LWP on solaris 8 box

Discussion started by: solea

10. Filesystems, Disks and Memory

Memory usage in the box

Discussion started by: ST2000