It creates as many processes as possible. Many versions of unix are configured with a kernel variable called maxuproc or something like that. It is max processes that a non-root user can create. That is really the only protection and even with that a program like this is a nuisance. As fast as you kill a process, another takes its place.
To recover, as root, su to the user who is running the "while(1) fork();". root will be allowed to switch a root process to this user even if this bumps the number of processes past maxuproc. Now you have a shell running as the user. The shell cannot fork(), but it can exec(). So enter the command:
exec /usr/bin/kill -9 -1
Killing process -1 actually signals all processes owned by the user. This is documented on the kill(2) man page and this is required by
posix. If there are a lot of processes and system calls are preemptable and processes with real-time priority are running, this may not work. A second approach is:
exec /usr/bin/kill -STOP -1
The STOP signal, whose number varies from system to system, cannot be caught. It is used for job control and suspends the process. A suspended process cannot attempt to fork() but it continues to consume a process slot so no new process can take its place. Once all of the offending processes are suspended, then you can kill them off.
Shells often have a built-in kill command... if it can handle -1 are a process number, then you can use that. But shells often have built-in kills that choke on the KILLALL constant.
