About solaris memory management, you can read whatever has been written by Richard Mc Dougall.
About Oracle memory management, you can read whatever has been written by a member of tha "oak table" (Tom Kyte, Jonathan Lewis, Tanel Poder...)
Still regarding oracle : read about PGA automatic Tuning (from 9i i guess) as well as SGA automatique Tuning (from 10g i guess) (get info about how to set parameters sga_target, pga_aggregate_target, workarea_policy...)
Read also the concept guide Oracle as well as the Oracle Tuning guide of the version you are running.
make sur you have not exceeded the number of process/sessions (see gv$resource_limit view)
make your your system is correctly sized (semmns semmni etc... solaris parameter should be sized accordingly with the oracle instance it hosts (see Oracle Recommandation).
If you run Oracle in a local zone on solaris 10 then make sure you have a very recent patch update (Solaris 10 (8/07)) because the proc_lock_memory privilege (check with ppriv -v <pid>) has been added by default in the non-global zone so you can take advantage of the dism (
Dynamic Intimate Shared Memory).