Shell Programming and Scripting

Actually it now mostly makes sense. Notice the comment after the line in question.

Code:

ls -l 2>&1 >&3 3>&- | grep bad 3>&-    # Close fd 3 for 'grep' (but not 'ls').
#              ^^^^   ^^^^
#              1111   2222    3333

The item which I numbered 1111 closes fd 3, but the ls command had already made use of fd 3. So the process, which will become the ls command, uses fd 3, closes fd 3, and then forks. The child process, which will become grep, springs into existence with no fd 3. Why the author put in sequence number 3333 is not totally clear. But we are not really guaranteed that the grep process will be a child process of the ls process. If the shell was to be rewritten so that the grep process begat the ls process we would be closing fd 3 late. So while sequence 3 is unneccessary today it might be a hedge against a future revision to the shell or to make it more portable. Another possible reason for the superfluous sequence 3333 is to make it totally clear to the reader that grep process has no access to fd 3.

But in any event, the comment was meant to pertain to sequences 1111 and 2222. That's why the line is there.

With all due respect to your seniority and all the great, informative posts you have contributed long before I was even a gleam in the forum's eye, I believe your analysis of this pipeline and its redirections to be off the mark.

I will be using "the main shell" to refer to the shell tasked with interpreting the ls-grep pipeline.

The grep process will never be a child of the ls process. And, the ls process will never be a child of the grep process. They will be siblings, each a child of the main shell. Further, the order in which they are created does not matter.

For grep to beget ls would require more than a shell rewrite, it would require a grep one too.

Actually, the fork will happen before the i/o redirection, then an exec. If the i/o redirection were done before forking, file descriptor handling would get very very messy, as the parent would have to juggle things around before the fork, and then restore sanity afterwards (assuming the parent has further business to tend to).

As I understand it, the chain of events to realize the ls-grep pipeline is as follows:

The main shell will first create the descriptors that the pipe will use. Then, it will fork twice (creating two subshells). One of the subshells will eventually exec to become ls and the other will eventually exec to become grep.

One subshell will modify stdout to point to the pipe (this subshell will become ls) and the other will modify stdin (for grep). Before exec, these subshells have to process each command's command-line i/o redirections. In the case of the subshell that will become ls, this would make fd 1 a copy of fd 3 and would then close fd 3. In the case of the subshell to become grep, there's nothing to be done but close fd 3. Now, when each subshell exec's, there is no fd 3 available to either process.

Agreed. The same is true of ls.

If I'm the one that's off the mark, I await the Cluestick of Enlightenment

Regards,
Alister

Not really true. I actually said that grep was the child and ls the parent. You made a slight goof and reversed my comments. But an experiment shows that the reversed statement you attributed to me is true. So I hereby adopt it as what I meant to say.

Code:

$ /bin/sleep 60 | grep bad &
[1]     32429
$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
xxxxx     3417  3416  0 Mar08 pts/1    00:00:00 -ksh
xxxxx    11703  3417  0 Mar08 pts/1    00:00:00 ksh
xxxxx    32424 11703  0 13:42 pts/1    00:00:00 ksh
xxxxx    32429 32424  0 13:43 pts/1    00:00:00 grep bad
xxxxx    32430 32429  0 13:43 pts/1    00:00:00 /bin/sleep 60
xxxxx    32431 32424  0 13:43 pts/1    00:00:00 ps -f
$

The grep process wound up as parent and the sleep process wound up as child. I would not want to bet that it always works that way. And no I don't have a special grep process that knew to establish the child.

32424 forked to create 32429 but both processes are still ksh because neither has exec'ed yet. 32429 then forked again to create a child 32430. At this point the whole pipeline exists and both processes exec'ed their respective programs.

If I recall correctly, only the AT&T ksh and zsh run the last command in a pipeline in the current shell environment.

Edit: Irrelevant, see further comments.

Only if that command is a shell built-in. I did not know that zsh does that too. Now that bash supports co-processes this issue is the primary reason I don't want to switch to bash.

It's interest that bash is making the main shell the parent of each process in the pipeline. I guess it needs this to support it's PIPESTATUS concept. This must make hard to get the entire pipeline into a separate process group.


BTW, regarding the original question, my admittedly flawed, reasoning still might be what the author of that example was thinking. If not, then I not know what he was thinking. Anyone got a better idea on that?

Yes, of course, thanks for pointing it out.

I stand corrected on my statement that grep and ls will be siblings. They usually are, but its an unprescribed implementation detail. I think my analysis is generally sound, though.

For kicks, I tested your sleep-grep pipeline on a few shells (old versions on a disused laptop); for each shell, I ran the pipeline in the background and in the foreground. att ksh93 backgrounded was the only one of six scenarios which showed that parent-child child relationship.

att ksh93: bg: parent-child
att ksh93: fg: siblings
bash: bg: siblings
bash: fg: siblings
pdksh: bg: siblings
pdksh: fg: siblings

Regards,
Alister