I've thought about this before, and it's tricky...
For starters I've often believed things like libc, libssl, libpam, and so-on ought to be kept in memory to improve reaction times under idle or high-load conditions. "not recently used" doesn't understand how some rarely-used things are important, and I'd certainly be willing to trade .1% of my disk cache to guarantee that sshd can respond instantly in ideal conditions and work at all in worst-case ones.
The "sticky bit", set on shared libraries, used to be used for something quite close to this -- it would keep the executable in swap even when unloaded. But it's fallen into disuse, and Linux doesn't honor it at all.
Even a user application can map in a file and mlock() at least some RAM, though, so you'd hope it'd be an easy enough fix.
Now, the next problem is where is the kernel going to get memory buffers to support another socket? Hopefully it's smart enough to not completely spend its own memory, since that's not something easily changed.
Another problem is process tables. Often there's just not enough memory to run another process, so you log in, run 'kill -9 rameater', and there's not enough memory to run
kill. You can do an end-run around this by running
exec kill -9 rameater instead, but it's a double-edged sword because it logs you out.
The next problem is the login system. You can lock libpam into memory, but you certainly shouldn't lock in /etc/passwd and /etc/shadow. They might cause some delay logging in.
Ultimately, there may be no perfect fix. This situation is the sort of thing you'd use a hardware watchdog-timer for.
---------- Post updated at 11:24 AM ---------- Previous update was at 11:22 AM ----------
Quote:
when completely out of control as you mention ping may even not reply...
Out of curiosity, why would ping not answer? The kernel's still in control even when the system's swapping madly.
