Linux

No these are developers running client circuit testcases thru simulation. They can get quite large. but on this one machine i want it restricted.

Perhaps what you're looking for is as simple as kernel params

I don't know if these will have the desired effect for you or not, but its worth a try. You can set these with the sysctl command on the fly, and they can also be placed in your /etc/sysctl.conf for persistence. See "man sysctl" for more info than a dullard like me can give you.

Parameters for shared memory
kernel.shmmni: maximum number of shared segments, system wide
kernel.shmall: maximum shared memory, system wide
kernel.shmmax: maximum shared memory segment size

Hope this helps.

Currently, there is no possible way to limit rss on a RHEL4/5 system. This is a known issue upstream. The best option would be limit as.

There isn't a good way to implement this, as it opens up for the possiblity of DoS attacks if you set the RSS low enough. Anything that uses glibc will have a good chunk of RSS dedicated to glibc, which is going to hurt them. There is also the question of what to do in NUMA situations.