Shell Programming and Scripting

Try adding the following in the beginning of your scripts:

Code:

ulimit -n 65535
ulimit -u 532480

The values are from the output of ulimit -a that you reported.
Just to make sure that your scripts run with correct settings when invoked by crond.

Ok, but I think that is not the issue. (Since I am even getting this from command line)

The frequency of the issue seems to be increases.
I am now getting the error even from command line. ( ls, grep etc )
On an average, 1 out of 10 commands are failed on command line. If I retry instantly, it succeed in 2-3 hits.

Additionally, I am getting this issue for all the users ( not just root).

You're hitting a resource limit. How many processes are currently running?
Please post the output from:

Code:

cat /proc/loadavg
cat /proc/sys/kernel/threads-max
cat /proc/sys/vm/max_map_count

Here is the output.

Code:

# cat /proc/loadavg
14.84 64.55 195.84 6/9452 13798

# cat /proc/sys/kernel/threads-max
1064960

#  cat /proc/sys/vm/max_map_count
65536

#

I'm out of ideas ...
I suppose that when you get those errors, you're hitting the max number of running processes (entities) limit,
even though the output above doesn't show that,
or, at least not, when you executed the cat command.

It would be easier to check this by setting the resource limits in your scripts
explicitly (as shown above with:ulimit -n/u).
If you never exceed those values, the scripts should run fine.

You said that sometimes you're getting the same errors while executing commands
in the shell: make sure that the same limits are currently set.

This sounds like you might have a process that is starting children, but not reaping their status when they die. When they die, they will release the memory they were using, but you won't be able to start more processes running with the same user ID until the number of processes being run by that user falls below the system limit for that user. And the number of processes being run includes those unwaited for zombies.

Again, you must run

Code:

ps -eL

or

Code:

ps -efL

to see the threads!
Linux uses clone not fork, and the ulimit -u is a thread limit - even if man page and other description say processes.
I have seen bad java applications cloning thousands of threads in a loop.