Shell Programming and Scripting

Hi. I'm having trouble writing a wrapper script for the command line text editor gnu ed.

I want to be able to run the following algorithm on ed:

1. Display, on stdout, a command which I intend to feed into ed (before I actually feed it to ed).
2. Actually feed the command into ed's stdin.
3. Wait until ed finishes displaying whatever stderr and stdout it must, redirecting all (stderr and stdout) to stdout.
4. End if the command is q or Q.
5. Otherwise, go back to step 1.

I want to achieve this, in order to be able to automate ed (or any other command-line utility for that matter) through bash scripts, WHILE displaying all i/o interaction (this is important).

Anyway, my first try was to use the following shell script

display_and_feed.sh :

Code:

#!/bin/bash

while read LINE ; do
        echo ${LINE} 1>&2
        echo ${LINE}
done

as follows

Code:

$ printf ",p\nh\na\nhello\nworld\nhow are you\n.\n,p\nh\nh\nQ\n" | ./display_and_feed.sh | ed

the problem being, that I get this messy, disordered output (which consists of a mix of stderr and stdout, by the way):

Code:

,p
h
?
a
Invalid address
hello
world
how are you
.
,p
h             <-- command 'h' shows before output of ',p'
hello
h
world
how are you
Invalid address
Invalid address
Q

...so I'm thinking, "maybe an ugly time delay will do...", so then I write a new script:

display_and_feed_with_delay.sh :

Code:

#!/bin/bash

while read LINE ; do
        echo ${LINE} 1>&2
        sleep 0.5
        echo ${LINE}
        sleep 0.5
done

...I run:

Code:

$ printf ",p\nh\na\nhello\nworld\nhow are you\n.\n,p\nh\nh\nQ\n" | ./display_and_feed_with_delay.sh | ed

And sure enough, the (SLOW) script, returns the orderly output:

Code:

,p
?
h
Invalid address
a
hello
world
how are you
.
,p
hello
world
how are you
h
Invalid address
h
Invalid address
Q

So I unsuspectingly cheer and celebrate, but when I try to store it all neatly in a file, I get:

Code:

$ { printf ",p\nh\na\nhello\nworld\nhow are you\n.\n,p\nh\nh\nQ\n" | ./display_and_feed_with_delay.sh | ed ; } >file 2>&1
$ cat file
,p
?
h
Invalid address
a
hello
world
how are you
.
,p
h                   <-- where did the output of ',p' go?
Invalid address
h
Invalid address
Q
hello             <-- Oh... THERE it is...
world
how are you

Apparently, stderr gets redirected to file before stdout does. I am puzzled. Why does this happen??

By the way, this also happens when I try to use a pipe instead of redirecting to a file:

Code:

$ printf ",p\nh\na\nhello\nworld\nhow are you\n.\n,p\nh\nh\nQ\n" |  ./display_and_feed_with_delay.sh | ed |& cat
,p
?
h
Invalid address
a
hello
world
how are you
.
,p
h                   <-- ...once more...
Invalid address
h
Invalid address
Q
hello             <-- ...stdout shows only after ed ends.
world
how are you

Is this an exclusive ed behavior, or do all programs first pipe their stderr and then their stdout? Is there a way to merge (interwine, in the correct order) stderr and stdout, for ed, in a case like this?

I mean, this behavior is kinda funny, because, when I run stuff like

Code:

$ { echo "1" ; echo "2" 1>&2 ; echo "3" ; } |& cat

I get

Code:

1
2
3

Not

Code:

2
1
3

By the way: it would be awesome if there was a way to tell when ed is expecting new input, so that I can signal some script, which sends input into ed, when the moment is right, to feed ed more commands, instead of using some ugly sleep 0.5 time delay. Is there any way to do this?

There are a number of wrong presumptions in your post about how some Unix-facilities work. I will try to address them one by one. Bear with me, sometimes the relation of explanation and problem will not immediately become clear. It will (should) make sense in the end.

You have to picture a Unix process like a garden hose. You put something in on top and it comes out at the bottom. If you connect two such processes with a pipeline symbol you stick two of these garden hoses together, so that the outcome of one goes directly into the other.

Now the picture of the garden hose is simple, but yet a bit misleading: first off, because processes change the data stream they are working on: you put lots of blue water in on top and get a small amount of red water out on bottom (or vice versa). But the more important shortcoming of the picture is that the garden hose is Y-shaped: it has two outputs, STDOUT and STDERR (and even more outputs if you defined them, but these two are there per default).

Notice that there is a strict difference between the outlet of a garden hose and the place it is directed at. You can catch what comes out of the hose by either this bucket or the other (or, see above, even the intake of another garden hose), but there is still a difference between the destination a hoses ending is pointed at and the destination itself.

Coming back to your script from this rather theoretical explanation: a command like "2>&1" means: "direct what comes out of stderr (one hose ending) to where the stdout (the other hose ending) is already pointing at". This does in no way imply that these two streams are now somehow "merged". They are not. They are just pointing to the same bucket, but are still two different hose endpieces. If you first redirect stderr to point to the same bucket (file or pipeline) as stdout and then redirect stdout to point to another bucket stderr will not be affected in any way - two different streams.

Now for another misconception directly related to your last point with the redirected "echo"s: in fact it is in no way guaranteed to get any particular succession if different streams are involved. If you write:

Code:

(echo a; echo b) | cat

you are guaranteed that "a" is displayed before "b". If you have "c" and then "d" coming out of stderr and you merge these two streams in one destination you are also guaranteed that "c" shows up before "d", but there is no such guarantee that "c" or "d" will show up in any particular position relatively to "a" or "b". it could be "a-b-c-d" or "c-d-a-b" or even "c-a-d-b" or "a-c-b-d" - all these variations would be valid ouputs for the following code:

Code:

(echo a; echo b; echo c >&2; echo d >&2) 2>&1 | cat

The reason is that stderr and stdout are radically different even if they look just like two of the same kind: one of them is buffered, the other one unbuffered. That means that repeated calls to output something are stored and then handled in one big rush versus processing each request as soon as it comes.


Another thing you seem to miss is a rather small issue: "sleep" can only handle integers. "sleep 0.5" might work on some system if this system has extended the standard but will definitely fail on most. If you want to "sleep for less than a second" i suggest you search this board for the "nodoff" utility Perderabo has programmed once.

I hope this helps.

bakunin

This bit...
...inspired me to search for "make stdout unbuffered bash" in google, which led me to a stack overflow thread, which in turn referred me to the expect command, which pretty much solves my original problem.

Thanks for the buffering details of stdin and stdout, bakunin, I appreciate it.