Shell Programming and Scripting

I have application which to the heavy stdout and I have diverted the stdout to log file.

this log file is writing very heavily and we have a script which rotates the logs.
logic for rotation is smthing like
cp logfile logfile.1
cat /dev/null > logfile

this logic was working fine till we had lesser load coming in logfile but Now there is much stdout going on and after doing a
cat /dev/null > logfile

logfile becomes of size 0 but again shows the same size as was it before.


so what I'm having is
logfile 1GB
after rotation
logfile.1 1GB
logfile 1GB & still appending the data.


I see the FD 1 for the log write is showing the same size as of the file and not coming down.

can you see any issue with this kind of log rotation why it is breaking on heavy IO ( my assumption is that FD 1 is still in blocking mode )
and any suggestion how it can be solved by scripting or othere logic/commands.


Thanks..
Navin

Yes this is a common issues, if a process is appending to the log when you truncate the applications file pointer is still at the end of the file. All sorts of weird things can happen (eg file shows 0 size from commands like ls but application still writes to the old file and uses up space on your filesystem).

Quite a few programs support kill with a HUP signal to get them to rotate their own log files (check the manual or doco for you app). If it dosn't have any built-in log rotate support, your only choice is to stop/suspend the application rotate logs and restart.

is there any other way to get this work around.

I know that same issue happens in Apache logs but Apache overcomes it with piped logging.

What exactly is that, if it is a use of pipes to get the data move around. can that be simulated here.

application where these logs are generating is a live system and I cant stop/suspend that.

any idea would be helpful.

Thanks
Navin

The apache program supports redirecting it's logfile output via a standard unix pipe to any command you like. Apache ships with the logrotate command that basically keeps a timer and after a certain amount of time (or bytes written) it closes it's output file and starts a new one.

Talking about pipes, another idea might be to replace the logfile with a named pipe see this Redhat Page for details. This looks a bit tricky and is likley to introduce a whole new set of issues.