How can i Globally set the maximum core memory a single process can take. IE, i want to set that no single process may get more than 11GB.

I am running red hat enterprise unix 4.

Don't know if Red Hat has this command, but see if it has "ulimit"

On solaris, it is the -v parameter

-v maximum size of virtual memory (in kbytes)

ulimit is specific to bash on red hat linux. and i don't need to limit virtual memory i need to limit actual memory usage by a task. And i also need to limit it globally, not on a task by task basis. Red hat has so far responded with the following:

There is an option to limit the rss usage by a process for a given user using /etc/security/limits.conf, but I found references that it does not work as expected.

I am escalating this to a senior engineer for further investigation, you should get an update soon.

Do you mean physical silicon memory rather than virtual spilled over into swap?

How big is your machine?

The machine is a dual opteron system with 12GB of ram and 20GB of swap configured. I want to prevent a single process from getting more than lets say 90% of ram. (maybe max around 10GB). I have one user running a development progrm which has grown to 14.3Gb on this machine which really killed the machines ability to do stuff.

14.3Gb

Is it a Java environment that is not running a garbage collector?

We had a similar kind of issue recently

but not 14.3 GB, that was around 3 GB

where there were construction of too many hash maps.
when drilled down to the solving the problem; we diagnosed that after using the hash components we made sure we deleted the contents in the hash one by one .

Well, that's a quite an easy one to address and resolve.