UNIX for Dummies Questions & Answers

I need to set up a strange system through which an arbitrary command is sent to a number of different servers (well, actually, VPS accounts). We have a command "vpass" that "passes" a command from the root level to resident VPS accounts. Suppose I wanted each VPS to do some trivial thing, like report its hostname. I might use a script like:

Code:

for account in `cat accountlist`
do
vpass $account hostname
done

So, as you can see, the vpass syntax is:

Code:

vpass accountname command__to_run

Now, it would be convenient to set up a little script that runs an arbitrary command that's in a file called "command". To replicate the results above, I create "command" with "hostname" in it, and then do:

Code:

cmd=$(cat command)

for account in `cat accountlist`
do
        vpass $account $cmd
done

That works just right. However, more complex commands fail. For example, this works:

Code:

for account in `cat accountlist`
do
vpass $account mysql --user=root --password=p@ssword -e "USE database ; SELECT stuff FROM table ORDER BY RAND() LIMIT 1;"
done

However, putting that exact MySQL command into the "command" file fails. If I add a line "echo $cmd" after the variable assignment, it gives exactly the right command back, yet passing it into the vpass command fails.

Any suggestions on where the failure might come from? I'm at a loss, mainly because of time-induced brain fuzziness. Many thanks in advance.


Update:

I added -x and checked the results... Apparently some strange things are being done to the command:

Code:

vpass acct1 mysql --user=root --password=p@ssword -e '"USE' database ';' SELECT stuff FROM table ORDER BY 'RAND()' LIMIT '1;"'

It seems to my naive troubleshooting that those changes-- the addition of single-quotes throughout the command-- must be the cause of the problem, unless I misunderstand the output of -x.

Update 2:

I don't know why this didn't occur to me before... lack of sleep? Anyway, xargs helps...

Code:

cat command | xargs vpass $account

So, that works, but I'm a little perturbed that the other method doesn't. If only for my own education, I'd appreciate any thoughts on why that's failing.

---------- Post updated at 10:36 AM ---------- Previous update was at 10:26 AM ----------

[/COLOR]And I just realized one really brain-dead simple solution: Let the server do the parsing. I do this over ssh sometimes.

Code:

 vpass $account bash < commands

This also prevents people from inserting clever things in the 'commands' file that end up running locally.

[rest of the post]

Your method is a little naive in a couple ways. For one thing, it's a useless use of backticks:

Code:

# bad -- a Useless Use of Cat and Useless Use of Backticks in one.
# It not only makes an extra process and wastes memory,  but
# might prevent you reading all of a file >64KB since it won't
# fit in one shell variable.
for LINE in `cat file`
do
        ...
done

# better.  Reads one line at a time with no extra process at all.
# Works on files of any size.
while read LINE
do
        ...
done < file

Also, your xargs solution could be done with one fewer process like

Code:

xargs vpass < command

Basically, whenever you do 'cat foo' you can replace that with a shell redirection in nearly all circumstances. This is quite a bit more efficient.

As for why the command isn't doing what you expect, the shell doesn't substitue strings more than once -- it won't check inside things it substitutes for things like quoted strings or variables. So embedding strings and variables in a string won't work unless you explicitly tell the shell to re-evaluate it, with eval. Note that eval supports all valid syntax so someone could inject variables or local commands where you didn't expect them by escaping $ and so forth.

Code:

#!/bin/bash
STR="\"this is a string\""
STR2='$HOSTNAME'

function args
{
  echo "Has ${#} arguments"
}

# it LOOKS right...
echo $STR
# But it's being split into "\"this" "is" "a" "string"!
args $STR
# This also looks right...
eval echo $STR
# eval tells it to re-parse the line after substitution.
eval args $STR

# This will not substitute in variables.
echo $STR2
# This will!
eval echo "$STR2"

[COLOR="#738fbf"]

Thanks for the reply and the excellent pointers. The methods suggested solve a number of problems. I wish it were otherwise, but my scripts probably have plenty of naive errors. I've gotten reasonably good at finding *a* solution to problems, but as I revisit old scripts (well, not really that old... I'm still pretty new at this, as is certainly clear...) I realize that they are far from optimal. I hope to continue discovering that, as that would mean that I'm learning better and better ways.

One general question... If I use the suggested methods:

Code:

vpass $account bash < commands

or

Code:

xargs vpass < command

does the shell have to re-read the contents of the "command" file each time, or is that somehow cached? This is a very small file, of course, with negligible read time; however, I suppose there might be cases in which re-reading could have an appreciable effect.

Again, thanks for the great reply.

I'd suggest against the xargs method. I'm surprised it worked at all for one thing; I didn't know it was able to swallow quotes like that, and I don't think you can depend on it always doing so the way you expect. Consider what xargs is actuallly for: converting lists into arguments. You're sending it scripts, not lists, which it may cheerfully mangle on the assumption that it can break lists apart wherever it wants.

Code:

echo a b c d e f | xargs cat

xargs might run cat a b c d e f. It could also run cat a b c ; cat d e f and not be in error. For cat, the result is the same. But you can't split a script like that and expect it to work!

vpass $account bash < commands on the other hand can swallow and run any valid shell script. You could put quite complex things in commands if you wanted.

As for performance, the shell has to read the file whenever it wants to run it, just like anything else. When you were running 'cat commands' it was reading it from disk too. The operating system will try to keep the entire file in memory if anything uses it, though, which lets the second time its read avoid the disk entirely. The OS will cache almost all disk I/O from any source if it can. (sometimes even when you don't want it to )

Your disk can probably read scripts faster than your computer can run them in any case.

On the remote host it's never cached. That's kind of the point -- you wanted to keep one master copy and not worry about having two dozen copies on your clients. This isn't a problem unless your script's gigantic, your network's very slow, and your connection's very unreliable.

The total disk reads and network traffic is similar to your original method anyway. The lines have to get from disk to A to B one way or another.