Shell Programming and Scripting

I tried ps -efL when you suggested previously. I believe, threads and process are different from each other?
I have seen huge count when did ps-efL. Most of them was having same PID (true by definition of threads) . I am confused whether the system process limit applies to threads also?

Yes, We have lots of java applications running on the systems. When we restarted those applications, the system gets stable for sometime (we get comparatively, less number of fork errors).

Does that mean those java applications are the culprit?


I had seen zombie/defunct processes, but didn't find much of them. They are not so many in count.

Rule of thumb: if there are more than 100 threads per process, then the Java process is badly programmed - talk to the application owner/vendor.
If there are more than 1000 threads per process, then the Java process is definitely wrong.
I suggest to put a "ulimit -S -u 3000" soft limit into each Java start script.
This is a per-user limit. Therefore, it makes sense to start your Java apps as different users.
Sometimes it also helps to limit the file handles with "ulimit -S -n 1024", the default on most Linux systems. (Some application vendors suggest to tune this up - with a negative effect.)
Last but not least I think (have not yet evidence) kernel.pid_max not only limits the processes but also the application threads.
Tune it up with "sysctl kernel.pid_max=99999", and also add this to /etc/sysctl.conf.
99999 i.e. 5 digits is safe with processes like xfs that have a buggy "pidfile" handling. BTW an immediate tune-up is safe; a tune-down should happen only in /etc/sysctl.conf to be applied at next system reboot.