Login or Register to Ask a Question and Join Our Community


Solaris

Zones and memory resource control issues

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Solaris Zones and memory resource control issues
# 1  
Old 12-08-2010
Question Zones and memory resource control issues
Hi all,

This is a cross-post from the Sun/Oracle forums (I would include the URL here, but the forum doesn't allow me), cos quite frankly, this forum seems to be more active...

I am maintaining an in-house Sun/Oracle x86 server (x4275) running Solaris 10 with zones for testing and development purposes. Basic specs: 2x4Core Xeon CPU & 72GB RAM. More detailed specs at the end of this mail. The global zone isn't running anything of relevance. The other zones are all running a combination of our own products together with Sybase ASE 15 servers. Our own products are a mix of C++ and Java (both stand-alone java processes as well as App Servers/JBoss). I've also got a single instance of VirtualBox running in the global zone just to play around with.

I'm looking for some assistance on properly setting up (shared) memory resource controls on this server, because we currently have issues with this.

After we installed the 5th or 6th zone, we started getting (shared) memory allocation errors. Processes fail to start with “mmap failed” errors, Java processes won't start due to ‘Error occurred during initialization of VM\n Could not reserve enough space for object heap”, Sybase fails to start due to shared memory allocation failures. If we take another zone down, we are able to start these processes in the first zone (but then we can't start them in the zone we just took down).

I can't really figure out why though. The server itself doesn't seem to be out of memory at all. Based on vmstat output it looks to me as if there's enough memory free still, and that there's hardly any swap space used:
Code:
# vmstat
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s1 s2 s3 s4 in sy cs us sy id
0 0 0 2723604 15251616 14 51 0 0 0 0 0 0 0 10 11 13099 67426 8142 2 1 98

That looks like 15GB free still to me.

Because of the Sybase shared memory requirements, I've (tried) to use Solaris projects to set resource controls on “system” on the global zone:
Code:
# cat /etc/project
system:0::::project.max-shm-memory=(privileged,107374182400,deny)
user.root:1::::
noproject:2::::
default:3::::
group.staff:10::::

Individual zones have a default /etc/project file, except for one zone, which has group.staff:10:::Smilieroject.max-shm-memory=(privileged,10737418240,deny) (all applications run under user accounts that are part of staff).

/etc/system doesn't have any memory related entries in it (just set rlim_fd_cur=1024; set rlim_fd_max=8192, cos I don't think you can set that with project controls yet).

Any suggestions for things to check out?

Kind regards, Maarten
Code:
==== Processor Sockets ====================================

Version Location Tag
--------------------------
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz CPU 1
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz CPU 2

Memory size: 73720 Megabytes (72GB)

# zpool status
pool: pool1
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
pool1 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
c0t5d0 ONLINE 0 0 0
c0t6d0 ONLINE 0 0 0
c0t7d0 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t0d0s0 ONLINE 0 0 0
c0t1d0s0 ONLINE 0 0 0

errors: No known data errors
# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
pool1 5.44T 891G 4.57T 16% ONLINE -
rpool 278G 9.18G 269G 3% ONLINE -

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
pool1 740G 3.72T 197G /pool1
pool1/zone7 51.6G 3.72T 51.6G /pool1/zones/zone7
pool1/u_boekhold 15.8G 3.72T 15.8G /pool1/home/boekhold
pool1/zone1 47.7G 3.72T 47.7G /pool1/zones/zone1
pool1/zone2 99.5G 400G 99.5G /pool1/zones/zone2
pool1/zone3 87.5G 3.72T 87.5G /pool1/zones/zone3
pool1/zone4 143G 881G 143G /pool1/zones/zone4
pool1/zone5 97.5G 3.72T 97.5G /pool1/zones/zone5
rpool 11.2G 262G 34K /rpool
rpool/ROOT 7.18G 262G 21K legacy
rpool/ROOT/s10x_u8wos_08a 7.18G 262G 7.18G /
rpool/dump 2.00G 262G 2.00G -
rpool/export 44K 262G 23K /export
rpool/export/home 21K 262G 21K /export/home
rpool/swap 2G 264G 16K -

# zoneadm list -vc
ID NAME STATUS PATH BRAND IP 
0 global running / native shared
4 zone3 running /pool1/zones/zone3 native shared
5 zone6 running /pool1/zones/zone6 native shared
7 zone7 running /pool1/zones/zone7 native shared
9 zone8 running /pool1/zones/zone8 native shared
10 zone4 running /pool1/zones/zone4 native shared
11 zone2 running /pool1/zones/zone2 native shared
12 zone1 running /pool1/zones/zone1 native shared
13 zone5 running /pool1/zones/zone5 native shared

(zones zone6 and zone8 are also supposed to be on separate nested ZFS filesystems, but whoever installed these made a mistake... will need to correct that later. Using nested ZFS filesystems so I can put a quota on each).

Any help appreciated,

Kind regards, Maarten
Last edited by Scott; 12-10-2010 at 02:26 PM.. Reason: Please use code tags
# 2  
Old 12-08-2010
What says
Code:
swap -s

?
# 3  
Old 12-10-2010
Quote:
Originally Posted by jlliagre
What says
Code:
swap -s

?
Sorry for the delay, have been out of the office...

Code:
# swap -s
total: 38506512k bytes allocated + 11278672k reserved = 49785184k used, 3538980k available
# swap -l
swapfile             dev  swaplo blocks   free
/dev/zvol/dsk/rpool/swap 181,2       8 4194296 4194296

Not sure how to read the output of swap -s, but the output of swap -l seems to tell me that I have a swap space of (only) 2GB, and that nothing of that is used.

I don't really understand the "available" column from swap -s: that only shows 3.5G available if I count correctly. If currently 47.5GB is in use on a 72GB server, surely there should be around 20-25GB space left?

Rgds, Maarten
# 4  
Old 12-10-2010
2 GB of swap space is likely too low for your virtual memory usage, especially for 72 GB of RAM server. I would suggest you to increase the swap size on your system. eg:
Code:
swap -d /dev/zvol/dsk/rpool/swap
zfs set volsize=16g rpool/swap
swap -a /dev/zvol/dsk/rpool/swap

Make sure your swap is not currently used with "swap -l" before running these commands.

For a detailed RAM usage display to see where the remaining 20-25 GB are used (likely in ZFS related kernel stuff), you can run that command:
Code:
# echo ::memstat|mdb -k

Last edited by jlliagre; 12-10-2010 at 07:49 AM.. Reason: add commands for the swap size increase to be effective without reboot.
# 5  
Old 12-10-2010
Hi,

Just added 32G swap space. Let's see how that turns out (after the weekend though).

that memstat|mdb command gives me (after adding the swap space):
Code:
# echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     834125              3258    4%
ZFS File Data             2074033              8101   11%
Anon                      9626639             37604   51%
Exec and libs              490864              1917    3%
Page cache                 281713              1100    1%
Free (cachelist)           912592              3564    5%
Free (freelist)           4650073             18164   25%

Total                    18870039             73711
Physical                 18362748             71729

How do I translate that into english? I assume that "Anon" is the memory used by my applications. The freelist seems to indicate to me that there's still 18G available though...

If there is still 18G free, how should the swap space become involved in this already? Don't get me wrong, I'd be thrilled if just adding some swap solves my issue, but I'd like to explain to my colleagues why that's necessary. I was the one insisting on dumping as much memory as possible into this server, and I would feel a bit silly now if it turns out we could just have created huge swap as well Smilie

Btw. I created the extra swap on my rpool, which is a mirror of 2 300GB SAS drives. My other pool is a raidz1 array of 6 1TB SATAII drives. Any reason why I should prefer one pool over the other for swap space?

Rgds, maarten
# 6  
Old 12-10-2010
Quote:
Originally Posted by boekhold
Just added 32G swap space. Let's see how that turns out (after the weekend though).
As I edited the command while you were replying, just make sure you delete/add the swap for the change to be taken immediatly.
Quote:
How do I translate that into english? I assume that "Anon" is the memory used by my applications. The freelist seems to indicate to me that there's still 18G available though...
Indeed, Anon is basically including all memory allocated by processes at run time (heap, stack and the likes).
Quote:
If there is still 18G free, how should the swap space become involved in this already?
Because there are different in concept. RAM is what you physically install in your box but (virtual) memory is what processes are living with. The virtual memory is the sum of a part of RAM and the whole swap area(s). If that sum is too small for all the memory reservations to fit, you'll have serious trouble. That doesn't mean you swap space will be used at all in term of disk I/Os. It is just that space must be here "just in case" it is needed later.
Quote:
Don't get me wrong, I'd be thrilled if just adding some swap solves my issue, but I'd like to explain to my colleagues why that's necessary. I was the one insisting on dumping as much memory as possible into this server, and I would feel a bit silly now if it turns out we could just have created huge swap as well Smilie
Both are fine. Adding RAM allows your applications and your OS to run at optimal speed, adding swap allows more of your applications run concurrently. If you have not enough RAM, your applications performance will likely degrade very significatively. If you have not enough swap, your applications either won't start or randomly crash ...
Quote:
Btw. I created the extra swap on my rpool, which is a mirror of 2 300GB SAS drives. My other pool is a raidz1 array of 6 1TB SATAII drives. Any reason why I should prefer one pool over the other for swap space?
As long as the RAM is large enough for the swap not to be used (swap -l), you can use the slower disks for swap. On the other hand, if your system start thrashing because of RAM exhaustion, the faster disks would be (slightly) better ...
# 7  
Old 12-10-2010
Quote:
Originally Posted by boekhold
Hi,

Just added 32G swap space. Let's see how that turns out (after the weekend though).
Decided not to wait and try it out immediately. Adding swap space seems to have solved by issue. The zone that we had to stop earlier could be started again now, with all its applications.

Strange thing is that even after starting that zone, there is no actual swap space used!

Code:
# swap -l
swapfile             dev  swaplo blocks   free
/dev/zvol/dsk/rpool/swap 181,2       8 4194296 4194296
/dev/zvol/dsk/rpool/swap1 181,3       8 67108856 67108856
#

See? blocks == free...

Is Solaris doing something like "eager ahead of time allocation", in the anticipation that it might need that memory in the future? Otherwise I have no explanation for this myself...

Maarten

---------- Post updated at 16:41 ---------- Previous update was at 16:22 ----------

Quote:
Originally Posted by jlliagre
As I edited the command while you were replying, just make sure you delete/add the swap for the change to be taken immediatly.
Actually I just added a second swap device (swap -l now has 2 lines output). Seemed to be the less risky option.

Quote:
Because there are different in concept. RAM is what you physically install in your box but (virtual) memory is what processes are living with. The virtual memory is the sum of a part of RAM and the whole swap area(s). If that sum is too small for all the memory reservations to fit, you'll have serious trouble.
The point for me is that I don't see any memory reservations that would exceed my physical RAM. Well, actually, that's not entirely true. Going back to the "memstat | mdb" output:

Code:
# echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     834125              3258    4%
ZFS File Data             2074033              8101   11%
Anon                      9626639             37604   51%
Exec and libs              490864              1917    3%
Page cache                 281713              1100    1%
Free (cachelist)           912592              3564    5%
Free (freelist)           4650073             18164   25%

Total                    18870039             73711
Physical                 18362748             71729

I see that Total is larger than Physical. I just don't understand why that would be the case. Couldn't Solaris "make this fit" by reducing the freelist a tiny bit in size? Why does the above list of zfs/anon/exec/page/freecache/freelist have to add up to something that exceeds physical RAM size?

Maarten
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Solaris

Solaris 11.2 x86 Local Zones for Oracle RAC database, intermittent Memory corruption of db

Hi, I was wondering if any Solaris fellow out there has dealt with running Solaris 11 x86 on Local Zones for Oracle RAC (which may or may not be relevant to issue), running on HP gen h/w. Every so often could be weeks between issues or some times days, there will be a memory corruption and db... (2 Replies)
Discussion started by: crossmypath
2 Replies

2. AIX

AIX 6.1 Memory Performance issues

Good Day Everyone, Just wonder anyone has encounter AIX 6.1 Memory Performance issues ? What I have in my current scenario is we have 3 datastage servers (Segregate server and EE jobs - for those who know Datastage achitect) and 2 db servers(running HA to load balance 4 nodes partitions for... (3 Replies)
Discussion started by: ckwan
3 Replies

3. Solaris

Check Solaris Zones Processor & Memory allocations

Hi, I am a newbee in the solaris administration. My question is how to 1. Check the total CPU and memory of a global zone. 2. Check the allocated CPU and memory for each of the residing non-global zones. I have already tried prtconf which gives the following output /usr/sbin/prtconf... (4 Replies)
Discussion started by: poga
4 Replies

4. Solaris

[DOUBT] Memory high in idle process on Solaris 10 (Memory Utilization > 90%)

Hi Experts, Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing. Why memory utilization on solaris always looks high? I have statement about memory on solaris, is this true: Memory in solaris is used for... (4 Replies)
Discussion started by: edydsuranta
4 Replies

5. Linux

Memory issues iin CentOS release 5.3 (Final)

New to the forum. I am running CentOS release 5.3 (Final) and have run into a unique situation. I have been able to determine that we have a swap memory leak due to an issue with an asterisk server. This specific issue takes approximately 2-3 months for enough of the swap to to be used before... (1 Reply)
Discussion started by: g0neinsane
1 Replies

6. Solaris

resource control in zones

I 'm getting following error while setting deny action for a zone.cpu-shares resource control prctl: unable to enable deny on rctl with global flag 'no-deny' and when i check with rctladm its shows me bash-3.00# rctladm zone.cpu-shares zone.cpu-shares syslog=n/a ... (0 Replies)
Discussion started by: fugitive
0 Replies

7. Solaris

Memory management in zones

whats the difference between setting zone capped-memory from zoncfg and setting rctl: name: zone.max-locked-memory .. if changed the zone.max-locked-memory with prctl it does not change in rcapstat .. but if change with rcapadm it reflects in rcapstat o/p (0 Replies)
Discussion started by: fugitive
0 Replies

8. Solaris

How to find Total and Free Physical Memory and Logical Memory in SOLARIS 9

Hi, Im working on Solaris 9 on SPARC-32 bit running on an Ultra-80, and I have to find out the following:- 1. Total Physical Memory in the system(total RAM). 2. Available Physical Memory(i.e. RAM Usage) 3. Total (Logical) Memory in the system 4. Available (Logical) Memory. I know... (4 Replies)
Discussion started by: 0ktalmagik
4 Replies

9. AIX

AIX-specific memory issues?

Hi, Does anyone here write cross-platform applications to be run on different Unix variants? For some reason, an application I'm working with, when being run on machines with similar specifications, I run into "cannot allocate" errors (St9bad_alloc) on AIX but not on other platforms. Is it a... (8 Replies)
Discussion started by: wolfier
8 Replies
Login or Register to Ask a Question