UNIX for Advanced & Expert Users

The shell is a more direct interface to the environment and the kernel than a lot of people realize. File numbers in shell are the file descriptor numbers the kernel has actually assigned to this process, nothing more, nothing less. Environment variables (though not non-exported variables) are expected to be put in a specific spot of memory and organized in a very specific way, to allow non-shell things to use them. This has certain problems as you've noticed. It also means you can use the shell as really low-level setup for something else, as there's no nonsense in the way of you opening whatever files you want as whatever descriptors you need. Kind of the original purpose of the shell -- setting up for something else.

Moving away from the model the kernel presents you would make it a more capable language, and a less capable shell.

The same is true of file descriptors in C - but when you call open() to open a file there's no chance of accidentally closing a file you had open - the newly-opened file is assigned to a previously unallocated file descriptor.

So there's that basic problem of syntax when using redirection syntax to open files... That much could be solved pretty easily. (Actually, I guess Bash already has it pretty much licked: "exec {newfd}<filename"... $newfd takes on the newly allocated file descriptor number...)

But why stop there? There's no reason the shell couldn't tie the idea of an "open file" (within the shell) to a specialized, non-exportable variable, which offers the possibility of scoping and reference-counting the file descriptor, closing it when it's unused within the shell process. This would also give us a natural way to express whether the file descriptor should be "exported" to child processes (i.e. whether it's kept open through a fork()/exec()) - if the variable is exported, the child process gets an environment variable with a file descriptor number in it, and that file is open. If it's not, the file descriptor is closed on (or before) exec().

There's no reason numeric redirections couldn't still be made available for those cases where you need to control exactly where an open file ends up for a specific child process. I think that's just not the common case. The common case is that people want to read from or write to files. The bash/zsh syntax that allows opening a file and assigning its descriptor to a variable is a step in the right direction, but I think there's no reason things couldn't be improved further.

I want to point out that exception because it's an important one. Variables implemented in shells are already removed from the reality of the underlying mechanism of environment variables. You've got integer variables, array variables, hash table variables... Shells provide these special cases despite the fact that they don't fit the underlying environment variable mechanism (and, therefore, can't be exported) because they're useful.

Well, granted, though with caveats.

Let's suppose, for instance, I actually were proposing a shell design that wouldn't allow for numeric redirection, and that I were running one of the few programs for which that feature is actually useful. (I only know of a few - xterm, screen - can't think of any others...) Then, yes, something of value was lost.

But this doesn't mean that working with raw resources must be the default means of interacting with the shell - only that it must be available, and that the shell's normal mode of operation shouldn't be too far removed from this. The raw control should be there when you need it, and out of the way the rest of the time.

The only time the shell knows it's done with a file descriptor is when it quits! It's not psychic... If it doesn't give you a minimal model of the kernel's I/O, I don't think you could call it a shell. If you don't want to use a shell language, don't use a shell language. A great deal of underlying assumptions of the process model -- stdin/stdout and by extension all pipes and redirection -- depend on the model we have. Don't ruin things for the rest of us because you want contradictory properties...

these is again depends on the user he prefers but - easy t ore-call last any commands instead you type they all from start - ksh -o vi

Wrong. The other time it knows it's done with a file descriptor is when that file descriptor is closed.

So why should "exec $fd<&-" be the only way to tell the shell it's time to close a file descriptor?

That's all I'm describing here - just another way to control the lifetime of an open file descriptor in a shell. The shell doesn't need to be psychic in order to know when a local variable goes out of scope, or when a global variable is unset. That's all you need for the RAII method of resource management.

If an open file descriptor in the shell is tied to the lifetime of a variable binding - what is the scenario in which that file needs to be open, but isn't because the shell decided it was time to close it?

It seems to me you have an extraordinarily limited view of what qualifies as a shell.

And, you know, I don't expect you're the only one. When I look at the problem of trying to enhance the shell (and I understand that what I consider "enhancement" others may very well consider garbage) - it seems likely to me that most of the people who actually still use the shell, probably don't want to see it change in any significant way. If they wanted to use something else, they probably already would be.

Nevertheless, I want to try to make something better.

I want to make a better shell language. I like the basic approach of the Unix shell but I think there's room for improvement in the execution. I mean, seriously - do you think the existing shell is perfect? Really?

Obviously, there is a chance that anything I come up with won't be an improvement. It may not even be worth running at all. Who knows? But we don't get nice things if people don't try to strike out in a new direction once in a while.

I really don't understand why you think anything I've suggested would change that model, or break those assumptions.

But don't worry, I won't ever break into your machine and replace your shell. :P'

Which is the model we have already. Your point? A few good reasons:

1) Right now there's no "special" files in a shell, you can do as you please even with stdin/stdout/stderr. Adding new features that act in different ways would have to work around these basic three if you want "echo asdf | something" to still work, making the language much less orthogonal.

2) Making opening, closing, and inheriting files more complicated makes the whole process of piping data between commands more complicated since it uses the same files and mechanisms. (Unless, as above, you add special cases for compatibility, which makes the language a mess instead). This is supposed to be easy in a shell.

3) You're ignoring perfectly good existing features. The "exec" opening methods and explicit file descriptor numbers often aren't necessary at all. How's this for a scoped file descriptor that closes itself?

Code:

while read LINE
do
        echo "${LINE}"
done <filename

My point is that adding scoped lifetime management to file descriptors in a shell wouldn't fall nearly as far outside the current model as you suggest.

So what's the problem? I don't see how anything I've described impacts file descriptors 0-2 or pipelining in general.

By "between commands" you mean between external commands, i.e. programs on the PATH, right? As opposed to shell functions, shell extensions, etc...

When the shell hands a file descriptor to a process (on exec()), it doesn't have to care what that process does with the FD. The underlying mechanism is as it always has been: the file occupies a numbered slot in the new process's file descriptor table.

It can be hard to come up with a good syntax when you're trying to serve multiple needs. I don't believe there's any reason this feature would make a mess of the syntax, however. If the variable $f is a "file" object (i.e. an open file descriptor, assigned some numeric FD within the shell itself which the user actually doesn't need to know...)

Code:

read x < $f   #Is that quite simple enough?  Alternately, "read x <&$f" would make it plain that $f is an open file and not a filename.  A shell could support either syntax, or both.
some_command 8<&$f   #And there's no reason it couldn't work with the existing numeric redirection syntax either...
some_command $f   # It's actually not too hard to make this do what the user intends...

There's a couple ways the last one could be handled:

First, the shell could handle it like process redirection: assign the file $f some numeric FD in the new process, then in place of the argument $f, substitute /dev/fd/(some number)
Second, it could be as above but with just the numeric FD rather than the /dev/fd path. Considering how few command-line tools actually provide any support for doing something with a numeric FD other than 0-2, it probably doesn't matter - though the first approach would work with more commands since /dev/fd/* can be used like any other filename...
Alternately, the shell could reject this as a type error. It's an easy enough condition to detect, and the shell could even provide a useful error message, explaining that you have to use redirection syntax to pass an open file descriptor to a process.

Honestly, the whole /dev/fd thing in process substitution always struck me as kind of hokey... Though that may just be because I hadn't heard of /dev/fd before (/dev/tcp struck me as even more hokey - in large part because the directory doesn't exist...) But it does its job, supplies the identity of an open file descriptor to a command being called as if it were a filename...

I think that takes care of "scoped", as you say - though the syntax could be better (you increase the nesting level each time you add an additional file, unless you open all the files in the same place... And if I'm not mistaken, the files can only be opened at the end of the block (I tried it at the beginning of a block, in bash, and it didn't work), which is kind of awkward IMO. When reading code I think it's more useful to be able to see what file is what at the point you start using them.

It's fine to point at an example where you're just reading one file, or reading one file and writing another, and say you don't need to deal with numeric FDs in the shell. But if you're opening more than one for reading, or you have different subroutines that are all operating on different files, then you certainly do need to think about where those files are going to go.

Bash already provides a syntax for auto-assigning a file descriptor when opening a file, of course - and that goes a long way toward making the situation nicer: you can open the file, pick an FD for it, and assign the number of that FD to an environment variable all with a compact bit of redirection syntax. That's about what you get from using file descriptors in C - the programmer takes responsibility for not overwriting the variable containing an important file descriptor and also takes responsibility for closing file descriptors when it's time. It's quite workable, really. Going C++ style with RAII and so on isn't strictly necessary, it just strikes me as something that could be useful.

But, of course, the proof is in the implementation. Whether or not an idea sounds good, specifics of execution can make all the difference.