Shell Programming and Scripting

I realize that you are aware that your suggestion isn't elegant, but aside from that its utility is very limited. It may work for that small sample, but it won't scale. In this case, the delay is long enough for the system to copy the ls stderr writes to a kernel buffer and for ls to move on and flush its stdout stream before tee gets a chance to write to its stdout. But, if there were a lot of error messages -- e.g. from many non-existent files -- the delay wouldn't be long enough. Also, you couldn't simply extend the delay without limit because eventually an ls stderr write() would block and no matter how long you waited, the ls stdout stream would not be flushed before ls stderr messages flow downstream through tee to grep.

The following is a much better workaround:

Code:

{
    exec 3>&1
    ls iexist idont 2>&1 1>&3 | tee -a /dev/null | perl -lpe 'BEGIN {$|=1;}'
} | cat > logfile

The trace confirms that it's not just luck or timing; the writes are indeed coalesced into full lines:

Code:

# 4271: tee (since tee does not buffer, it appears to mimic GNU ls's write behavior)
# 4272: perl
4271  write(1, "ls: ", 4) = 4
4271  write(1, "cannot access idont", 19) = 19
4271  write(1, ": No such file or directory", 27) = 27
4271  write(1, "\n", 1)                 = 1
4272  write(1, "ls: cannot access idont: No such"..., 51) = 51

There are undoubtedly small utilities out there to absorb a stream and increase the level of buffering, but I have never bothered to search for them.

From the ugly-solution-dept: The following works, but it depends on too many unreliable implementation details to be a reassuring solution:

Code:

# ls (GNU coreutils) 8.13
# GNU bash, version 4.2.24(1)-release (x86_64-pc-linux-gnu)

{
    exec 3>&1
    ls iexist idont 2>&1 1>&3 | tee -a /dev/null | 
    {
	while IFS= read -r linebuf; do
	    printf '%s\n' "$linebuf"
        done
    }
} | cat > logfile

I definitely recommend the perl solution over a sh/bash hack, because there's nothing preventing the printf builtin from being fully-buffered when writing to a pipe. The correct solution to this problem requires line-oriented buffering (of course, of lengths <= PIPE_BUF).

Regards,
Alister

Of course, I saw this. But I though: "you work more, you just need to sleep a bit more..." . Ah ah, LOL.

This is what I didn't think of. I see it now. Thanks.
--
Bye

Fantastic! This is one of the threads that keep me coming here again and again.

When i wrote my first post i had the following script as data source for tests (AIX 6.1 last TL):

Code:

#! /bin/ksh

print -u1 "This goes to stdout."
print -u2 "This goes to stderr."

exit 0

I understand now why that worked with my redirections but in other more real-life situations they might fail.

My focus is on writing scripts to accomplish tasks and i want my own messages (error and warning/info) to be as tolerant as possible, so here is what i suggest as a solution for writing custom scripts:

- Put a time stamp in every line of output. Even if the lines in the output file will become disorganized, a simple "sort" will put them in order again. I use a certain "output format" for my lines, which is consistent across scripts:

[PID TIMESTAMP MSGCLASS message]

where MSGCLASS is either "Info", "Warning" or "Error".

- Work similar to "syslog": all (error/warning/info) messages to a info log, and error messages also to a separate error log. This way you can avoid the elaborate redirection gymnastics for the usual things you want to achieve.

I hope this helps.

bakunin

Hi bakunin,
thank you for your notes! I tried your adjustments, but the STDOUT will always displayed on screen.

I need this for a cronjob. Actually everything (STDOUT & STDERR) is written to a log. But I want to get informed, when an error occurs.

As it is, alister did most of the work and you should thank him.

hmm, that is unexpected. With the example script i gave (see up this thread). I was able to get the result you wanted, albeit we learned from alister that it was more out of chance and won't work in larger scales.

A cron job has no terminal attached to it at all and output to <stdout> is usually mailed to the owner of the cron job. This is the reason why <stderr> and <stdout> in cronjobs are always redirected - you don't want to get all these mails.

If you want output to go to the "system console" (don't confuse this with a terminal - the console can be any terminal, but not every terminal is the console) use syslogs facilities instead of simple output. Syslog messages can be configured to go either to the system console or every terminal. An example for this would be the "shutdown" command, which usually prints a "The system is about to go down"-message on every terminal. This is done via a syslog facility.

I hope this helps.

bakunin