Shell Programming and Scripting

Great! You know, UNIX utilities you use in shell scripting are behaving like a fine-tuned orchestra: it is possible to make them sound awful but let the right conductor enter the stage and they will create true magic. It is fun to write well written programs like it elating to witness or perform any other piece of art.

Now, to your problem:
The problem is in this line and it is something that will certainly cross your path more often in the future. It is an ability (as well as a fundamental behavior) of the shell, which is called field splitting.

Let us start with a simple question. Consider the following command:

Code:

# ls -l /some/directory/file.txt

We will certainly agree that ls is a command, -l is an option and /some/directory/file.txt is the filename (and the argument to the command). But why is this so? Wouldn't it be equally possible that -l /some/directory/file.txt is the filename and there are no options? Or that ls -l is the commands name?

In fact it is not: the reason is that the three distinct parts are separated by a blank and therefore constitute three separate "words". The shell takes the first word as the commands name, which is why ls gets invoked. Then it passses the other two words as separate entities to ls and this program then knows which to take as option and which as argument. But the basis for this was that the shell has split the input to three words first.

You may have come across the "IFS". This stands for "internal field separator" and is a variable which contains the character(s) the shell uses to separate fields. Per default this is set to "blank" and ""tab" (the characters, not the strings). It is possible, though, to set these to other values.

You can use this whole mechanism to your advantage, i.e. in a read-statement: read can not only read input and assign one variable but many. Here is a little example program that illustrates that:

Code:

#! /bin/bash

one=""
two=""
# first we set up an input file to use:
echo "first word" > /tmp/myinput
echo "second line" >> /tmp/myinput
echo "third and last line" >> /tmp/myinput

# Now, this is not very impressive, right?
while read one two ; do
     echo "$one" "$two"
done < /tmp/myinput

# but this is exchanging the words, showing what the two variables really hold and adding visible delimiters:
while read one two ; do
     echo "\"$two\"" "\"$one\""
done < /tmp/myinput

exit 0

You may notice that in the last line of the file the last variable in the read-statement gets all the "leftover" words if there are fewer variables than words. This is intentional. You can also change the IFS so that what constitutes a "word" changes:

Code:

#! /bin/bash

one=""
two=""
# first we set up an input file to use:
echo "first word" > /tmp/myinput
echo "second line" >> /tmp/myinput
echo "third and last line" >> /tmp/myinput

# we change the IFS so that words are not split along blank-boundaries any more:
while IFS="," read one two ; do
     echo "\"$two\"" "\"$one\""
done < /tmp/myinput

exit 0

Because no "," was found in the whole input every line consisted of only one word which went to "$one" and "$two" remained empty.

Now, coming back to your problem: You have input that looks like this:

Code:

apples : 1111 : aaaa
pears : 2222 : bbbb
peaches : 3333 : cccc

Because the IFS is set to blanks per default it is split at blanks into words by the shell and this is why you get everything as a separate menu entry. It would be logical to simply set the IFS but i have tried it and it doesn't work - don't ask me why, i don't really understand bash. But there is another way: let us say you have a file called: file with a blank in its name. What would you do if you have to ls it:

Code:

# ls -l *name
total 196
-rw-rw-r-- 1 bakunin bakunin    35 Sep 12 08:26 file with a blank in its name

# ls -l file with a blank in its name
ls: cannot access 'file': No such file or directory
ls: cannot access 'with': No such file or directory
ls: cannot access 'a': No such file or directory
ls: cannot access 'blank': No such file or directory
ls: cannot access 'in': No such file or directory
ls: cannot access 'its': No such file or directory
ls: cannot access 'name': No such file or directory

We need a way to switch off the shells field splitting. Fortunately there is one: quoting. Quoting means to surround a string by quotes (single or double quotes). This does some other things too, but it also turns off field splitting:

Code:

# ls -l "file with a blank in its name"
total 196
-rw-rw-r-- 1 bakunin bakunin    35 Sep 12 08:26 file with a blank in its name

This is why most shell programmers are adamant about quoting, like i quoted the strings in the example programs above: it makes sure the script works even if there is a string with a blank or tab in it. Alas this won't work in your case either because you do not have fixed strings but generate the arrays contents in a subprocess dynamically. You can of course change the last sed-statement to surround the output with quotes (i took the liberty to remove the superfluous grep because sed can do that too):

Code:

options=( $(awless -l list routetables --columns ID,NAME,VPC --filter VPC=vpc-0025xxxxxxxe52 | sed -n '/rtb/ {;s/|//g;s/^/"/;s/$/"/;p;}') )

And there is another way you probably have figured out already and wonder why i haven't mentioned it all the time (well, actually because i wanted to show you all the alternatives): you can use a while-loop to read in the array and if you only use one variable all the content of the line ends there. Notice the quotation at the assignment of the array elements, though, otherwise you get a syntax error:

Code:

while read LINE ; do
     array[((${#array[@]}+1))]="$LINE"
done <<EOF
$(awless -l list routetables --columns ID,NAME,VPC --filter VPC=vpc-0025xxxxxxxe52 | sed -n '/rtb/ s/|//gp')
EOF

You sure ask yourself why this instead of:

Code:

awless -l list routetables --columns ID,NAME,VPC --filter VPC=vpc-0025xxxxxxxe52 | sed -n '/rtb/ s/|//gp' |\
while read LINE ; do
     array[((${#array[@]}+1))]="$LINE"
done

This is one of the great shortcomings of bash and one of the reasons why i prefer Korn shell: the second process of a pipeline (which is in this case the while-loop) is executed in a subshell, therefore the variables defined there are local to that subshell. If you want to use the cotent of array outside of this you need to use a here-document, a subshell and some redirection.

OK, finish for today. I hope it helps and you enjoyed it.

bakunin