Login or Register to Ask a Question and Join Our Community


Linux

Reason/components behind high memory utilisation on linux

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Linux Reason/components behind high memory utilisation on linux
# 1  
Old 08-31-2012
Reason/components behind high memory utilisation on linux
Hi friends,
Problem: High memory utilisation of one linux server, due to which a database crashed on the server.

My question to all: How does glance calculate that memory utilisation on the linux server is 98%.
I mean what components are part of (31.4gb-599mb)? How can i check that myself? Can you please guid me the commands/steps by which i can do that.
I want to understand what is taking so much memory and why.

Below is the glance (memory) command output and few other command outputs for your reference

Code:
GSTOMS2 /oravl01/oracle> uname -a
Linux indlin706 2.6.18-238.1.1.el5 #1 SMP Tue Jan 4 13:32:19 EST 2011 x86_64 x86_64 x86_64 GNU/Linux


GSTOMS2 /oravl01/oracle> glance
B0000A Glance 11.00.044         07:28:43 indlin706   x86_64                                                                         Current  Avg  High
------------------------------------------------------------------------------------------------------------------------------------------------------
CPU  Util   S     SU         U                                                                                                       | 15%   16%   17%
Disk Util   DD                                                                                                                       |  2%    3%    4%
Mem  Util   U                                                                                                                    U   | 98%   98%   98%
Swap Util   U                             U                                                                                          | 26%   26%   26%
------------------------------------------------------------------------------------------------------------------------------------------------------
                                                                    MEMORY REPORT                                                         Users=    5
Event         Current   Cumulative   Current Rate   Cum Rate   High Rate
--------------------------------------------------------------------------------
Page Faults     49036        54384      5980.0     5280.0      5980.0
Page Ins            0            0         0.0        0.0         0.0
Page Outs           0            0         0.0        0.0         0.0
Paged In          0kb          0kb         0.0        0.0         0.0
Paged Out         0kb          0kb         0.0        0.0         0.0
File Page Outs                           383.6
File Page Ins                            144.5

Phys Mem:  31.4gb Avail Mem:  31.4gb Free Mem:   599mb Buf Cache:   101mb
File Cache:   396mb


GSTOMS2 /oravl01/oracle> cat /proc/meminfo
MemTotal:     32959324 kB
MemFree:        217704 kB
Buffers:        102760 kB
Cached:         403864 kB
SwapCached:     421960 kB
Active:        7235420 kB
Inactive:      2083436 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32959324 kB
LowFree:        217704 kB
SwapTotal:    14680048 kB
SwapFree:     10921936 kB
Dirty:             232 kB
Writeback:         140 kB
AnonPages:     8753644 kB
Mapped:         123776 kB
Slab:           436868 kB
PageTables:    1329540 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  22771100 kB
Committed_AS: 30406876 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    270268 kB
VmallocChunk: 34359467763 kB
HugePages_Total:  8192
HugePages_Free:   3432
HugePages_Rsvd:    718
Hugepagesize:     2048 kB
GSTOMS2 /oravl01/oracle> ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x00000000 5668865    root      644        80         2
0x00000000 5701635    root      644        16384      2
0x00000000 5734404    root      644        280        2
0x06347849 5767173    root      666        65544      1
0x31023803 5832711    root      666        131164     2
0x00000000 8224776    oracle    640        4096       0
0x00000000 20873225   oracle    640        33554432   2750
0x00000000 20905994   oracle    640        4613734400 2750
0x449b7868 20938763   oracle    640        2097152    2750
0x00000000 1240203276 oracle    640        16777216   254
0x00000000 1240236045 oracle    640        1207959552 254
0xc6c9d620 1240268814 oracle    640        2097152    254
0x00000000 21495823   oracle    640        8388608    98
0x00000000 21528592   oracle    640        515899392  98
0x2cc69600 21561361   oracle    640        2097152    98
0x00000000 21626898   oracle    640        8388608    34
0x00000000 21659667   oracle    640        515899392  34
0xe6c4ff24 21692436   oracle    640        2097152    34
0x00000000 21757973   oracle    640        8388608    32
0x00000000 21790742   oracle    640        515899392  32
0x0e8e4708 21823511   oracle    640        2097152    32
0x00000000 21889048   oracle    640        8388608    93
0x00000000 21921817   oracle    640        515899392  93
0xc1e942f4 21954586   oracle    640        2097152    93
0x00000000 22380571   oracle    640        8388608    93
0x00000000 22413340   oracle    640        515899392  93
0x51a25230 22446109   oracle    640        2097152    93
0x00000000 22511646   oracle    640        8388608    95
0x00000000 22544415   oracle    640        515899392  95
0x45d7e150 22577184   oracle    640        2097152    95
0x00000000 22642721   oracle    640        16777216   127
0x00000000 22675490   oracle    640        1207959552 127
0x964bc9b0 22708259   oracle    640        2097152    127
0x00000000 23199780   oracle    640        16777216   32
0x00000000 23232549   oracle    640        1207959552 32
0x90afabcc 23265318   oracle    640        2097152    32
0x0103283e 7962663    root      664        4192       1
0x0103281b 7274536    root      664        4192       1
0x0103280c 7307305    root      664        4192       1
0x01032821 7372842    root      664        4192       1
0x01032825 7405611    root      664        4192       1
0x01032835 7503916    root      664        4192       1
0x01032829 7569453    root      664        4192       1
0x0103282d 7634990    root      664        4192       1
0x01032831 7897135    root      664        4192       1
0x0103283b 7929904    root      664        4192       1
0x00000000 8257585    oracle    640        4096       0
0x5d9c1fa8 8290354    oracle    640        4096       0

GSTOMS2 /oravl01/oracle> free -m
             total       used       free     shared    buffers     cached
Mem:         32186      31976        210          0        100        394
-/+ buffers/cache:      31481        705
Swap:        14335       3670      10665


GSTOMS2 /oravl01/oracle> top -c
top - 07:29:48 up 5 days,  1:22,  5 users,  load average: 2.39, 1.54, 1.33
Tasks: 3977 total,   3 running, 3971 sleeping,   0 stopped,   3 zombie
Cpu(s):  8.3%us,  2.8%sy,  0.0%ni, 86.3%id,  1.2%wa,  0.2%hi,  1.2%si,  0.0%st
Mem:  32959324k total, 32744588k used,   214736k free,   103756k buffers
Swap: 14680048k total,  3758100k used, 10921948k free,   405676k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11197 root      15   0  7040 3892  500 S  9.0  0.0 128:01.14 /opt/hped/Plugins/usage/discusge -d /var/opt/hped/.discagnt -n 365
 3110 oracle    15   0 15868 4284  840 R  6.5  0.0   0:01.70 top -s3 -d2 -c
 9239 oracle    15   0 1402m  22m  18m R  5.6  0.1  47:18.20 oracleGSTAPRM1 (LOCAL=NO)
18935 oracle    18   0 4694m  41m  13m S  2.5  0.1 192:56.50 ora_dia0_GSTABP1
20100 oracle    15   0 4666m  21m  18m S  1.2  0.1   0:33.21 oracleGSTABP1 (LOCAL=NO)
20917 oracle    18   0 1403m  19m  13m S  0.6  0.1   6:43.60 ora_dia0_GSTAMSS1
21282 oracle    15   0 4664m  19m  16m S  0.6  0.1   0:07.81 oracleGSTABP1 (LOCAL=NO)
21667 oracle    15   0 1403m  17m  13m S  0.6  0.1  27:22.19 ora_dia0_GSTOMS2
27662 oracle    15   0 4667m  28m  23m S  0.6  0.1   0:12.03 oracleGSTABP1 (LOCAL=NO)
32062 oracle    15   0 4667m  28m  23m S  0.6  0.1   0:07.17 oracleGSTABP1 (LOCAL=NO)
32709 oracle    15   0 4666m  22m  18m S  0.6  0.1   0:01.24 oracleGSTABP1 (LOCAL=NO)
  383 oracle    15   0 4664m  19m  17m S  0.3  0.1   0:01.08 oracleGSTABP1 (LOCAL=NO)
  613 oracle    15   0 4664m  19m  17m S  0.3  0.1   0:00.20 oracleGSTABP1 (LOCAL=NO)
 1610 oracle    15   0 4666m  21m  18m S  0.3  0.1   0:04.46 oracleGSTABP1 (LOCAL=NO)
 1676 oracle    15   0 4664m  19m  16m S  0.3  0.1   0:01.36 oracleGSTABP1 (LOCAL=NO)
 3293 oracle    15   0 4664m  19m  16m S  0.3  0.1   0:09.95 oracleGSTABP1 (LOCAL=NO)
 3297 oracle    15   0 4664m  19m  16m S  0.3  0.1   0:10.32 oracleGSTABP1 (LOCAL=NO)
 3299 oracle    15   0 4664m  19m  17m S  0.3  0.1   0:08.80 oracleGSTABP1 (LOCAL=NO)
 7916 oracle    15   0 4667m  27m  22m S  0.3  0.1   0:10.67 oracleGSTABP1 (LOCAL=NO)
 7966 oracle    15   0 4666m  23m  22m S  0.3  0.1   1:35.17 oracleGSTABP1 (LOCAL=NO)
 8603 oracle    15   0 4664m  19m  17m S  0.3  0.1   0:00.76 oracleGSTABP1 (LOCAL=NO)

GSTOMS2 /oravl01/oracle> ps -fe |grep -i pmon |grep -v grep
oracle    9872     1  0 Aug27 ?        00:00:22 asm_pmon_+ASM
oracle   18921     1  0 Aug27 ?        00:20:45 ora_pmon_GSTABP1
oracle   19686     1  0 Aug27 ?        00:00:38 ora_pmon_GSTCRM55
oracle   19869     1  0 Aug27 ?        00:00:26 ora_pmon_GSTCRM65
oracle   20037     1  0 Aug27 ?        00:00:24 ora_pmon_GSTCRM66
oracle   20234     1  0 Aug27 ?        00:00:41 ora_pmon_GSTCRM67
oracle   20488     1  0 Aug27 ?        00:00:36 ora_pmon_GSTCRM56
oracle   20681     1  0 Aug27 ?        00:00:39 ora_pmon_GSTCRM60
oracle   20877     1  0 Aug27 ?        00:00:38 ora_pmon_GSTAPRM1
oracle   20892     1  0 Aug31 ?        00:00:13 ora_pmon_GSTAMSS1
oracle   21653     1  0 Aug27 ?        00:00:23 ora_pmon_GSTOMS2
GSTOMS2 /oravl01/oracle> vmstat 3 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 3836244 182268 107468 407824   16   18   662   216    3   25 15  7 74  5  0
 2  0 3836244 182144 107500 407824    0    0    77   434 2260 9385  6  3 90  1  0
 0  0 3836244 182176 107508 407836    0    0   171   269 2140 9518  6  2 91  1  0
 2  0 3836244 186068 107520 407832    0    0    85   430 2282 9955  8  5 86  1  0
 1  0 3836244 196044 107544 407836    0    0    64   241 2017 9022  8  6 86  1  0

Please let me know if you want to know the output of any commands etc or any other information needed to understand this high memory utilisation issue more.

Thanks,
Kunwar
# 2  
Old 09-01-2012
Define crashed, please. What Oracle error? I think it is likely that you have heap fragmentation.

You apparently have a lot of db's running on this box. Up through 11g, oracle will not create non-contiguous shared memory segments for the "SGA" without a nasty warning. The db in question will start but you may get memory errors later on. I would look in the oracle alert logs for these warnings. Or ask the DBA to do it.

Heap fragmentation means: there are lots of allocated memory chunks sprinkled around, often with intervening smaller areas of free memory. While the smaller ones are perfectly usable some applications have issues with trying to use a bunch of small ones as a single large block. Oracle has this issue.

The simple solution is either more memory or move off db's to another box.

BTW: for linux showing lots of memory "in use" is completely normal. Your problem seems to be from what I see in the ipcs -m display.
# 3  
Old 09-01-2012
It is obvious that Oracle takes most of the memory available. When Oracle starts the DB it takes a configurable amount of memory, the SGA (system global area). It does its own sort-of file caching in there (DB block buffers, ...) as well as memory-intensive operations (indexing, ....). The SGA is so-called "pinned memory", that means it is not swapped out under any circumstance.

Apart from that Oracle has separate processes by which it communicates with the outside world: the "listeners". In your "top" output these are the processes with "(LOCAL=NO)" at the end. Every listener needs some memory too and this might add up to something. How many instances of a listener process are started depends on the load on the database (if there are less DB connections there will be less listener processes), but is also configurable somewhere in the DB startup files.

I have only some cursory knowledge about these things (I am a Sysadmin, not a DBA, after all), but your DBA can probably off the top of his head explain these things in detail to you. I suggest you set together with him/her and discuss these issues.

I hope this helps.

bakunin

/P.S.: Now that Jim mentions it, i believe he is right: the heap looks definitely fragmented and, yes, this can cause problems.
# 4  
Old 09-01-2012
Quote:
Originally Posted by jim mcnamara
Define crashed, please. What Oracle error? I think it is likely that you have heap fragmentation.
--Crashed means: Database instance was shut down. All the processes (Background and foreground) just went down.


You apparently have a lot of db's running on this box. Up through 11g, oracle will not create non-contiguous shared memory segments for the "SGA" without a nasty warning. The db in question will start but you may get memory errors later on. I would look in the oracle alert logs for these warnings. Or ask the DBA to do it.
--I am the DBA , but not a very experienced one. I checked the alert log, i dont see any memory related warnings before the crash or even now.

Heap fragmentation means: there are lots of allocated memory chunks sprinkled around, often with intervening smaller areas of free memory. While the smaller ones are perfectly usable some applications have issues with trying to use a bunch of small ones as a single large block. Oracle has this issue.
--One question. How i can confirm that there is indeed a heap fragmentation. Any commands to run or any specific logs/files to look at.

The simple solution is either more memory or move off db's to another box.

BTW: for linux showing lots of memory "in use" is completely normal. Your problem seems to be from what I see in the ipcs -m display.
--Sorry didnt understand. What part of ipcs -m you see as problematic. If it is what should have been the value for it to be fine?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

Swap memory usage is high in Linux

Hi , There is one following alert . Message : cdm:Average (2 samples) swap memory usage is now 91%, which is above the warning threshold (90%) Here is my findings. Output of TOP command in Linux server. top - 14:21:44 up 6 days, 4:48, 1 user, load average: 2.55, 2.06,... (3 Replies)
Discussion started by: Maddy123
3 Replies

2. Shell Programming and Scripting

Memory Utilisation of a server

Hi, I need to find the memory utilisation of a server as mentioned below. Memory should be divided into TOTAL, UZDM, BUFFER, CACHE, AVAILABLE. Can anyone help me to get this? (10 Replies)
Discussion started by: Arasu
10 Replies

3. Red Hat

Memory conmsumption by Java server is very high on Linux compared to Soalris

Hello All, We have a jave server running on both linux and Solaris environments. On solaris,it consumes only 600-700MB whereas it goes upto 21G in Linux. I am monitoring the memory consumption through top command. Is this high memory consumption expected in Linux? (1 Reply)
Discussion started by: prasperl
1 Replies

4. Red Hat

Memory Utilisation of httpd process

Hi All, I have a RHEL 5.4 server which runs an Apache web server(2.2.23 compiled with prefork MPM).I want to find out the memory utilised by the web server. # ps -ylC httpd | wc -l 245 # ps -ylC httpd | more S UID PID PPID C PRI NI RSS SZ WCHAN TTY TIME CMD S ... (0 Replies)
Discussion started by: Hari_Ganesh
0 Replies

5. AIX

CPU and memory utilisation by one process

Hi, We need to monitor one process over time to see at which exact point it starts to consume to much memory and CPU (to correlate this with some other events). I am a newbie in AIX, so please could you tell me which command will give a snapshot of the current state of a process so that I... (1 Reply)
Discussion started by: yallow
1 Replies

6. IP Networking

bandwidth utilisation high after implementing firewall

hi, we have one unix server which sends multicast packets in the network, recently we have implemented firewall in between unix server and clients.After implementing firewall we found our lease line utilisation has been increased dramatically. Is this because of firewall or else...please... (2 Replies)
Discussion started by: ajaysahoo
2 Replies

7. Shell Programming and Scripting

Start a new process when memory/cpu utilisation falls

Hi all, I am a newbie to whole shell scripting thing. I deal with computational studies and a lot of softwares which run on linux. At times, I have to schedule my jobs after some process has started. In these case, I have to use at command to schedule my job. However, I would like to know if... (0 Replies)
Discussion started by: johnsmithgr8
0 Replies

8. Red Hat

What would cause high kernel/system utilisation

We have a VM running RedHat that seems to have very high system (kernel) utilisation compared to our physical servers running the same OS and workload that show normal system time utilisation. Any ideas? Thanks in advance. (2 Replies)
Discussion started by: azurty
2 Replies

9. UNIX for Dummies Questions & Answers

Script for CPU and Memory Utilisation

Hi, I want to check the CPU and Memory Utilisation in the whole machine (not for a single process). Can someone send me a ready made script which captures all information in a log every 5min?? Thanks, Ajith (0 Replies)
Discussion started by: Ajith Praveen
0 Replies

10. Linux

High Mem & Cpu Utilisation

Hi All, Kindly help me in optimizing the server as it displays a great amount of CPU & MEM being utilised when the mysql process executes. Below are the stats --- -------------------------------------------------------------------------- # top 15:51:57 up 23:22, 5 users, load average:... (1 Reply)
Discussion started by: gautamatul82
1 Replies
Login or Register to Ask a Question