What is on Your Mind?

I'm actually not against that. As long as a variable always acts like a variable, the special cases I keep harping about don't exist. That's the whole point of polymorphism, yes? Lets you use different things the same way?

Perl does a really bad job of polymorphism. It's specialized itself into a big mess of special types and operators that are all context-sensitive. Depending on what kind of variable it is, print $variable, "\n" could print a string, a blank line, a mess of crap, the length of something, a pointer dump like ARRAY(0x123851023), a syntax error, or a host of other things. You can spend hours just trying to figure out what kind of variable someone's perl module hands off to you, let alone how to use it.

The shell has avoided this by keeping the meaning of its operators consistent and only having one interface for variables, strings. ${something} means string. ${#something} means length. ${*} means all arguments. "${@}" means all arguments without splitting on IFS. They work alone or in combinations, on arrays or strings or environment vars or local vars. The shell knows the difference and that's enough. In the few cases you're not allowed to combine them, that's always syntax error, not a silent substitution of nothing, not a length you didn't ask for, not a garbled mess, not a pointer dump. You can always expect to access a shell variable the same way no matter what it is, and the interface is flexible enough to encompass most possibilities. That's polymorphism.

And also why I think expanding the concept of variable instead of tacking the new concept of scoped files onto the side of it would work better. Let the variables be accessed the same way, as strings, since it's already convenient for the programmer, fits nicely with the existing syntax for redirecting files, and wouldn't need huge amounts of surplus new syntax to add. All that needs to be different about the variable is the ability to close a file when it goes out of scope. The scoping doesn't have to be something that causes the programmer to need new syntax to use, any more than a C programmer would use a stack variable differently than a global one.

---------- Post updated at 04:41 PM ---------- Previous update was at 03:52 PM ----------

Does that have to be done as part of the parallel loop? That sounds like something you might want to do in general, inside or outside a loop. Let the parallel loop be the parallel loop and the TTY sharer be the TTY sharer. That could be confusing if the threads end up doing different things on each iteration. I agree it could be useful but I'm not sure it should be part of the shell script. You end up with shell scripts that require expect to run, and start spewing garbage to stdout if you try to disentangle them. Utilities like ssh and su require password input to be from a terminal, but PTY's count. Something that gives you quick and easy access to PTY's, let alone in a high-performance parallel manner, isn't the sort of tool you want to just leave lying around -- another reason expect's a last resort. True, but take a look at the way these features are made. A shell script isn't going to blow up because someone disabled tab-completion -- shell scripts don't use tab completion, that's a user thing. Nor are they going to start typing in pgup to recall the last command and blow up when history features turn out to be unavailable, that's a user thing too. You cannot write scripts depending on these features and that's intentional -- the shell might care if it's in a terminal or not, but the program usually doesn't.

That's what I mean by not building it in, making it something like a debug flag -- to change the behavior of something to let it pop up these little windows. The script doesn't have to know or care whether you're snooping on its threads outputs or not, that can be left up to the shell. After all, what would know better about the current terminal than the shell?
It doesn't have to be all-or-nothing. I think it's possible to make a syntax that's both convenient and general-purpose. Not THAT complicated.

Code:

#!/bin/bash

THREADS=5       ;       WAIT=0  ;       END=0

# Wait for next thread.
function waitnext
{       # Needs BASH/KSH
        wait "${PIDS[$(( (WAIT++) % THREADS))]}"
}

for ((N=0; N<100; N++))
do
        [ "$((END-WAIT))" -ge "$THREADS" ] && waitnext
        sleep 1 & PIDS[$(( (END++) % THREADS ))]=$!
done
# wait for ALL remaining processes
wait

There's a shell feature missing that'd make this better and easier -- the ability to wait for any one process without saying which one. Right now you have a choice between waiting for one specific process or waiting for all of them. [edit] Or trapping SIGCHLD. That'd work, but would work better if shell had something like semaphores

That's just timesharing. If you want actual benefits from threading, you must do multithreading and/or multiprocessing. Kind of what i said but in more detail It is, and could cause other problems. If the parent closes a file descriptor, this child launcher will have to follow its lead somehow.

There's a better way to do this but I don't quite remember it. Something to do with fork callbacks. ...What? Really?

That's perfect for scoping your files! Allow local variables in braces like that. Your files could be some of these variables, containing FD numbers. When they go out of scope, the files close.

I'd considered that: the multiplexing/TTY sharing stuff would be useful outside the context of loops (or, if you prefer, at least as useful as it is in the context of loops. ) so addressing it as a general mechanism rather than something specific to loops would have advantages, make the shell language more orthogonal, etc.

One approach to doing this would be to have a separate step in which the PTYs and/or pipes are created to serve as /dev/tty, stdout, stdin for various jobs that are going to be run asynchronously: the file descriptors for these could be stored in array variables and passed to the loop construct, which would then pass them on to individual loop steps. Then probably one could come up with other cases where those FD arrays could be useful...

"expect" shouldn't be needed. Probably the sensible default for this "TTY sharing" stuff is to simply not do the TTY sharing thing if there's no TTY. ('course, there would be exceptions - cases where you might want TTY sharing to do its thing even if there's no TTY... If you're displaying the TTYs with xterms, for instance.)

Hm, so is the problem just that people could use this capability to spoof input to programs that interact on /dev/tty?

I can live with that. Anyone who wants to do damage with that kind of capability can do it regardless of whether the shell serves up the functionality or they have to write their own program for it.

Well, yeah, but my aim with "shell threading" is mostly just to keep things from getting booted into a "subshell context" simply because they're occupying a particular position in the command. But I'm starting to feel like it's probably not the best way to go. It'd mean keeping track of all the "threads" that are blocking on something, maybe stuffing all that into a big select() call...

Actually, though (I had to look this up) - it turns out when you fork() in a (Posix) threaded app, the new process has just one thread initially. The threads aren't cloned. So there'd be a little care necessary to make sure the environment's in a usable state prior to calling fork() but otherwise using Posix threads shouldn't be a problem.

I meant screen, sorry. Probably doesn't change the answer though. But where would the output go instead? Sure, if given access to them. Just something to keep in mind. External commands have to be in a seperate context. There's just no other way to run them. Builtins could be run in threads, so wouldn't need a separate context.

Even if you fork(), it's possible to share memory by other means. Memory you've created with mmap() can be shared between such processes. So just being in a subshell doesn't mean you'd have to lose access to variables. Environment variables though come preallocated so can't be shared that way without torturing your process environment in cruel and unusual ways.

Some builtins, like echo, might not need any context. When you know the amount of text is smaller than the size of your pipe, you know it'll never block -- so just cram the text in the write-end and close the write-end in advance. You don't need a new thread just to do that. Why not let the mutexes do the blocking? Reference, please? Things I've seen suggest quite differently. Then again, that's something I saw inside the linux pthreads library, so that might be things they had to do to prevent threads being cloned rather than things I'd have to do...

If true, that would make it a ton easier.

Another way to handle anonymous files could be similar to the awk way. When you redirect a file in awk, it stays open, referring to the same filename later just gets you the same handle over and over.

Code:

LINE=1
# Read lines from "otherfile" and "cmpfile" one by one, without
# redirecting the entire loop's stdin.
# Also set the close-on-exec flag for files opened in this fashion.
while read LINE <<"otherfile" && read OTHERLINE <<"cmpfile"
do
        if [ "$LINE" != "$OTHERLINE" ]
        then
                echo "Line $LINE doesn't match"
                echo "What would you like to do?"

                read RESPONSE
                case "${RESPONSE}" in
                *)   # todo: something
                      ;;
                 esac
        fi
        ((LINE++))
done

close "cmpfile" "otherfile"

Using << for 'keep the file open' seems a nice opposite of the >> 'append to file' redirection. This breaks the syntax for here-documents though. Not quite sure how to get the FD out of that either, maybe $<"filename" ?

Same place it would go to normally. Whatever the shell sees as FD #1.

Obviously if you're using stdout as some kind of formatted value stream, you don't want a bunch of processes writing data to it concurrently without some kind of synchronization: this is why I described a stdout sharing mechanism (separate from the /dev/tty sharing mechanism).

If somebody ran a multithreaded loop whose jobs produced newline-delimited value streams - that's a common enough formatting convention that it could just be built in to the shell. If those processes were writing out JSON or some XML schema, then it's something the user should probably be handling, by providing a program that takes those output streams and merges them into one.


Yeah, mainly by "things" I meant builtins like variable assignment or "read". Things you would do to modify the environment, in cases where it's pretty reasonable for the user to expect that it'll modify the main shell's environment...

Hm, might have to think about that one. Of course you could get around it by simply copying all the env. variables to the shell's process memory and operating on those copies of the variables - a bit wasteful but a pretty simple dodge...

Because I'm not talking about cases where threaded built-ins are blocking on some shared resource, I'm talking about them blocking on I/O, probably to a pipe as part of a larger job...

APUE chapter 12.9 explains the situation with pthreads and fork():
"Inside the child process, only one thread exists"
Of course I would be willing to bet there are some platforms or versions of platforms that got that wrong... APUE provides some coverage of different implementations to show differences but it'd take more time to tell you if it lists any for fork() in pthreads...

That's partway to just creating a new process then, since it would have some of the same side-effects -- changes from one wouldn't propagate back and vice versa.

Well, we're talking about using shared memory to export this copy of the shell variables to fork()'ed shell processes - so as long as the memory is shared writable, and as long as access to it is synchronized adequately, those shell processes would be able to propagate their variable changes back to the main shell...