Solaris

I had actually faced this issue with the process belonging to RMI service. I am trying to learn about the cause which triggered this scenario. Our server is basically running very large applications and very less manual intervention is required. So I suspect that there was any manual intervention in killing the process abruptly or the deletion of the files.

Can we also think that the application issues may also lead to this scenario. Also please share what precautions are required for preventing such issues.

Thank you.

The only way to get a file with a link count of zero is for one or more running processes to have that file open. Once all processes that had the file open have died, the blocks allocated to that file will be freed.

To get to the state that you're seeing, someone or something had to unlink the log file while the process that was writing to it was still running. The most common way that that happens is for someone to notice that a file is big and use rm logfile to remove it without killing the process that was writing to it. As has already been said in this thread; removing the last link to a file does not deallocate any blocks that have been written to that file. All allocated blocks will remain in the file until the last open file descriptor to that file has been closed.

If someone had killed the process, you might find files left around, but they would not have a link count of zero!

Depending on the flags used when the file was opened, where the file is written to by the writing process, and the specifics of the underlying file system, even truncating an open file isn't guaranteed to work.

And since redirected stdout and stderr tend to be done with ">>" pipe operators, the log files get opened with the O_APPEND flag with all writes going to the current file descriptor offset for the running process, truncations of such files rarely work. At best if the file system supports sparse files you will release the space by turning the log file into a sparse file.

Redirecting stdout and/or stderr for log files is a bad idea. The running process is tied to the log file(s) and the log file(s) are tied to the running process. You can't "age" the log files. You can't delete them. You can't even safely truncate them. Not only that,. using redirected stdout/stderr also causes all kinds of problems with log entry atomicity for multithreaded applications.

If logging is important, use a proper logging system designed for logging. Operating systems come with those.

Likely. As I already wrote, you should start by looking what your process is writing to stderr. Either there are a huge quantity of error or warning messages that need to be seriously investigated instead of being ignored or there are harmless informational messages sent there and in that case, you should check if a configuration wouldn't allow a less verbose output.
As already suggested by achenle, put in place a serious log process, with log rotation, archival and removal.

Note that truncating the log file (or its /proc zero link incarnation) will free disk space in your case as the underlying file system used is supporting sparse files with no doubt.

If it ws worth oogging, it may well be worth keeping and not losing, which can happen with truncate even if you make a copy in the window between the copy and the truncate.

Really nice log files have the date in their names and are changed at midnight, or the like. Some also change every time the log gets big. Then you can compress, archive or purge without stopping. Sometimes a helper app sends it a signal or it sets an alarm for itself, so you do not have to check the time with every log write. Then there is syslog(), where the logging to flat file is moved to a system daemon. There are a ton of posts on log rotation schemes and tools.