This is a cross-post from the Sun/Oracle forums (I would include the URL here, but the forum doesn't allow me), cos quite frankly, this forum seems to be more active...
I am maintaining an in-house Sun/Oracle x86 server (x4275) running Solaris 10 with zones for testing and development purposes. Basic specs: 2x4Core Xeon CPU & 72GB RAM. More detailed specs at the end of this mail. The global zone isn't running anything of relevance. The other zones are all running a combination of our own products together with Sybase ASE 15 servers. Our own products are a mix of C++ and Java (both stand-alone java processes as well as App Servers/JBoss). I've also got a single instance of VirtualBox running in the global zone just to play around with.
I'm looking for some assistance on properly setting up (shared) memory resource controls on this server, because we currently have issues with this.
After we installed the 5th or 6th zone, we started getting (shared) memory allocation errors. Processes fail to start with “mmap failed” errors, Java processes won't start due to ‘Error occurred during initialization of VM\n Could not reserve enough space for object heap”, Sybase fails to start due to shared memory allocation failures. If we take another zone down, we are able to start these processes in the first zone (but then we can't start them in the zone we just took down).
I can't really figure out why though. The server itself doesn't seem to be out of memory at all. Based on vmstat output it looks to me as if there's enough memory free still, and that there's hardly any swap space used:
That looks like 15GB free still to me.
Because of the Sybase shared memory requirements, I've (tried) to use Solaris projects to set resource controls on “system” on the global zone:
Individual zones have a default /etc/project file, except for one zone, which has group.staff:10:::roject.max-shm-memory=(privileged,10737418240,deny) (all applications run under user accounts that are part of staff).
/etc/system doesn't have any memory related entries in it (just set rlim_fd_cur=1024; set rlim_fd_max=8192, cos I don't think you can set that with project controls yet).
Any suggestions for things to check out?
Kind regards, Maarten
(zones zone6 and zone8 are also supposed to be on separate nested ZFS filesystems, but whoever installed these made a mistake... will need to correct that later. Using nested ZFS filesystems so I can put a quota on each).
Any help appreciated,
Kind regards, Maarten
Last edited by Scott; 12-10-2010 at 02:26 PM..
Reason: Please use code tags
Sorry for the delay, have been out of the office...
Not sure how to read the output of swap -s, but the output of swap -l seems to tell me that I have a swap space of (only) 2GB, and that nothing of that is used.
I don't really understand the "available" column from swap -s: that only shows 3.5G available if I count correctly. If currently 47.5GB is in use on a 72GB server, surely there should be around 20-25GB space left?
2 GB of swap space is likely too low for your virtual memory usage, especially for 72 GB of RAM server. I would suggest you to increase the swap size on your system. eg:
Make sure your swap is not currently used with "swap -l" before running these commands.
For a detailed RAM usage display to see where the remaining 20-25 GB are used (likely in ZFS related kernel stuff), you can run that command:
Last edited by jlliagre; 12-10-2010 at 07:49 AM..
Reason: add commands for the swap size increase to be effective without reboot.
Just added 32G swap space. Let's see how that turns out (after the weekend though).
that memstat|mdb command gives me (after adding the swap space):
How do I translate that into english? I assume that "Anon" is the memory used by my applications. The freelist seems to indicate to me that there's still 18G available though...
If there is still 18G free, how should the swap space become involved in this already? Don't get me wrong, I'd be thrilled if just adding some swap solves my issue, but I'd like to explain to my colleagues why that's necessary. I was the one insisting on dumping as much memory as possible into this server, and I would feel a bit silly now if it turns out we could just have created huge swap as well
Btw. I created the extra swap on my rpool, which is a mirror of 2 300GB SAS drives. My other pool is a raidz1 array of 6 1TB SATAII drives. Any reason why I should prefer one pool over the other for swap space?
Just added 32G swap space. Let's see how that turns out (after the weekend though).
As I edited the command while you were replying, just make sure you delete/add the swap for the change to be taken immediatly.
Quote:
How do I translate that into english? I assume that "Anon" is the memory used by my applications. The freelist seems to indicate to me that there's still 18G available though...
Indeed, Anon is basically including all memory allocated by processes at run time (heap, stack and the likes).
Quote:
If there is still 18G free, how should the swap space become involved in this already?
Because there are different in concept. RAM is what you physically install in your box but (virtual) memory is what processes are living with. The virtual memory is the sum of a part of RAM and the whole swap area(s). If that sum is too small for all the memory reservations to fit, you'll have serious trouble. That doesn't mean you swap space will be used at all in term of disk I/Os. It is just that space must be here "just in case" it is needed later.
Quote:
Don't get me wrong, I'd be thrilled if just adding some swap solves my issue, but I'd like to explain to my colleagues why that's necessary. I was the one insisting on dumping as much memory as possible into this server, and I would feel a bit silly now if it turns out we could just have created huge swap as well
Both are fine. Adding RAM allows your applications and your OS to run at optimal speed, adding swap allows more of your applications run concurrently. If you have not enough RAM, your applications performance will likely degrade very significatively. If you have not enough swap, your applications either won't start or randomly crash ...
Quote:
Btw. I created the extra swap on my rpool, which is a mirror of 2 300GB SAS drives. My other pool is a raidz1 array of 6 1TB SATAII drives. Any reason why I should prefer one pool over the other for swap space?
As long as the RAM is large enough for the swap not to be used (swap -l), you can use the slower disks for swap. On the other hand, if your system start thrashing because of RAM exhaustion, the faster disks would be (slightly) better ...
Just added 32G swap space. Let's see how that turns out (after the weekend though).
Decided not to wait and try it out immediately. Adding swap space seems to have solved by issue. The zone that we had to stop earlier could be started again now, with all its applications.
Strange thing is that even after starting that zone, there is no actual swap space used!
See? blocks == free...
Is Solaris doing something like "eager ahead of time allocation", in the anticipation that it might need that memory in the future? Otherwise I have no explanation for this myself...
Maarten
---------- Post updated at 16:41 ---------- Previous update was at 16:22 ----------
Quote:
Originally Posted by jlliagre
As I edited the command while you were replying, just make sure you delete/add the swap for the change to be taken immediatly.
Actually I just added a second swap device (swap -l now has 2 lines output). Seemed to be the less risky option.
Quote:
Because there are different in concept. RAM is what you physically install in your box but (virtual) memory is what processes are living with. The virtual memory is the sum of a part of RAM and the whole swap area(s). If that sum is too small for all the memory reservations to fit, you'll have serious trouble.
The point for me is that I don't see any memory reservations that would exceed my physical RAM. Well, actually, that's not entirely true. Going back to the "memstat | mdb" output:
I see that Total is larger than Physical. I just don't understand why that would be the case. Couldn't Solaris "make this fit" by reducing the freelist a tiny bit in size? Why does the above list of zfs/anon/exec/page/freecache/freelist have to add up to something that exceeds physical RAM size?
Hi,
I was wondering if any Solaris fellow out there has dealt with running Solaris 11 x86 on Local Zones for Oracle RAC (which may or may not be relevant to issue), running on HP gen h/w. Every so often could be weeks between issues or some times days, there will be a memory corruption and db... (2 Replies)
Good Day Everyone,
Just wonder anyone has encounter AIX 6.1 Memory Performance issues ? What I have in my current scenario is we have 3 datastage servers (Segregate server and EE jobs - for those who know Datastage achitect) and 2 db servers(running HA to load balance 4 nodes partitions for... (3 Replies)
Hi,
I am a newbee in the solaris administration. My question is how to
1. Check the total CPU and memory of a global zone.
2. Check the allocated CPU and memory for each of the residing non-global zones.
I have already tried prtconf which gives the following output
/usr/sbin/prtconf... (4 Replies)
Hi Experts,
Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing.
Why memory utilization on solaris always looks high?
I have statement about memory on solaris, is this true:
Memory in solaris is used for... (4 Replies)
New to the forum.
I am running CentOS release 5.3 (Final) and have run into a unique situation.
I have been able to determine that we have a swap memory leak due to an issue with an asterisk server. This specific issue takes approximately 2-3 months for enough of the swap to to be used before... (1 Reply)
I 'm getting following error while setting deny action for a zone.cpu-shares resource control
prctl: unable to enable deny on rctl with global flag 'no-deny' and when i check with rctladm its shows me
bash-3.00# rctladm zone.cpu-shares
zone.cpu-shares syslog=n/a ... (0 Replies)
whats the difference between setting zone capped-memory from zoncfg and setting
rctl: name: zone.max-locked-memory .. if changed the zone.max-locked-memory with prctl it does not change in rcapstat .. but if change with rcapadm it reflects in rcapstat o/p (0 Replies)
Hi,
Im working on Solaris 9 on SPARC-32 bit running on an Ultra-80, and I have to find out the following:-
1. Total Physical Memory in the system(total RAM).
2. Available Physical Memory(i.e. RAM Usage)
3. Total (Logical) Memory in the system
4. Available (Logical) Memory.
I know... (4 Replies)
Hi,
Does anyone here write cross-platform applications to be run on different Unix variants?
For some reason, an application I'm working with, when being run on machines with similar specifications, I run into "cannot allocate" errors (St9bad_alloc) on AIX but not on other platforms. Is it a... (8 Replies)