I am assuming you have a zone. If you are running out of heap for the jvm process, then there may be a zone cap on memory. Other settings can also affect this. It is also possible that you are actually running out of memory. So we have to go from square one.

From the global zone please post the output of (with jvm up)

Code:

prstat -Z
zonecfg -z [zone name here] info

from the zone itself please post:

Code:

swap -l