Shell Programming and Scripting

Here we go. The reason to use quotes round the range and and example of tr misbehaving on HP-UX if there are single character filenames in the current working directory.

Code:

echo "NEE"|tr [A-Z] [a-z]
nee
cd ..
echo "NEE"|tr [A-Z] [a-z]
tr: The combination of options and String parameters is not legal.
Usage: tr [ -c | -cds | -cs | -ds | -s ] [-A] String1 String2
       tr [ -cd | -cs | -d | -s ] [-A] String1
ls -lad ?
-rw-r--r--   1 root       sys              0 Jun  7 11:58 a
-rw-r--r--   1 root       sys              0 Jun  7 11:58 b
-rw-r--r--   1 root       sys              0 Jun  7 11:58 c
echo "NEE"|tr '[A-Z]' '[a-z]'
nee
echo "NEE"|tr 'A-Z' 'a-z'
nee
# And finally - protecting the square brackets from the Shell (which is the same as using quotes).
echo "NEE"|tr \[A-Z\] \[a-z\]
nee

P.s. My post yesterday was bad because I just happened to try the command in a directory containing single character filenames! However the HP-UX man tr does show ranges complete with square brackets and surrounded by double quote characters.

I was looking at the opengroup's tr documentation and indeed it mentions that System V tr did use square-brackets to delimit ranges; BSD systems did not.

The standard chose the BSD syntax as the lesser of two evils (most SysV scripts would continue to work fine, as opposed to breaking every BSD script using ranges).

More detailed info in the RATIONALE @ http://pubs.opengroup.org/onlinepubs...lities/tr.html

While the range syntax just discussed is specified clearly, the source of your syntax error is the result of undefined behavior. To do its job, tr requires the second of two character strings to be at least as long as the first. When the second character string is shorter (after expanding ranges, character classes, and repetition operations), BSD-ish tr pads the second string by repeating its final character until the string lengths are equal. This padding behavior does not allow the situation which triggered the syntax error to occur. SysV-ish tr does not pad and errors out.

Padding is discussed in the APPLICATION USAGE section of the opengroup tr manual page linked above.

I have no experience with HP-UX. While the results of your tr invocations seem to indicate that your tr is POSIX-compliant, the HP-UX tr manual I consulted states that ranges can be specified with and without brackets.

From http://h20000.www2.hp.com/bc/docs/su.../c02273397.pdf
If that's accurate, then it cannot be compliant as those are not equivalent expressions per the standard.

Expected result:

Code:

$ echo '[abc]' | tr '[a-c]' '[.*]'
.....
$ echo '[abc]' | tr 'a-c' '[.*]'
[...]

I would be curious to know the result of those commands on your system.

Regards,
Alister

@alister

Code:

echo '[abc]' | tr '[a-c]' '[.*]'
[...]
echo '[abc]' | tr 'a-c' '[.*]'
[...]

I can get the square brackets to translate with:

Code:

echo '[abc]' | tr '[a-c]\[\]' '[.*]'
.....

I think that this proves that square brackets are special in the syntax of tr range specifications. It is also the syntax I have used for umpteen years on assorted versions of unix. The slight variation on a Linux or BSD system is no surprise.

Finally this tr -d illustrates what I mean:

Code:

echo "[abc]" | tr -d '[a-c]'
[]

I use tr -d frequently in numeric or alphabetic data validation.
What you you get on a Linux system with that command?

Based on your results, HP-UX tr is not POSIX-compliant in ways that have been part of the standard for at least 15 years now (perhaps 20). I do not say this pejoratively; it's merely an observation.

I could not find IEEE Std 1003.2-1992 online, which I believe is the first standard to include the utilities (IEEE Std 1003.1-1988 only covered core system services).

The Single UNIX ® Specification, Version 2
Copyright © 1997 The Open Group
tr manual page
http://pubs.opengroup.org/onlinepubs...lities/tr.html


As I said before, I have no experience with HP-UX nor do I know what it aspires to be.

Perhaps backwards compatibility is most important to HP and its userbase. If that's the case, then it was a mistake to add support for the POSIX/BSD range syntax. a-c in historical SysV tr means three characters, a, -, and c; it's equivalent to ac-.

If, however, HP endeavours to be POSIX-compliant, then your results are unexpected and erroneous; scripts that are compliant and work as expected on compliant systems can fail on HP-UX.


That's the expected result for historical SysV behavior, but it's not POSIX-compliant. In a POSIX tr range expression, the brackets are not special at all; [a-c] is equivalent to ][a-c.

The POSIX-compliant result is .....



The \[ and \] escape sequences are undefined in POSIX. Their use is not portable.


That gives me nothing (except for the untranslated newline emitted by echo), which is the POSIX-compliant result.


Linux or BSD is irrelevant for the purposes of this discussion. I'm simply playing POSIX lawyer at the moment .

It appears that HP-UX tr added support for the BSD range expression syntax that POSIX long ago adopted, a-c, but it continues to accept historical SysV syntax, [a-c], treating them identically even though according to POSIX they mean different things (the latter includes two brackets which the former does not).

It's understandable that you've been using this syntax for a very long time without any obvious problems. With a SysV tr, the range expression behaves as you intend. With a POSIX or BSD tr, in most instances, where both strings consist of a range expression, the brackets are silently translated into identical characters. While the brackets were not intended to be members of the translation set, since they are translated into themselves, the result is correct (which is why the POSIX standard chose to go with the BSD syntax, less collateral damage). However, in other cases, for example, when only the first string contains a range expression and the second is a repetition expression, tr '[a-z]' '[.*]', there exists a potential for a silently erroneous result. And if the tr implementation does padding on the second string, then the repetition expression isn't required for a silent error to occur, tr '[a-z]' '.'.

methyl, I greatly appreciate your responses to my questions. I realize that these are rarely encountered corner cases, but they pique my curiosity. I often learn more than I intend as I dig into them.

Regards,
Alister

I can only explain the issue in post #1 if the O/P has a combination of file(s) in the current working directory which causes Shell to change the tr command to something wrong but syntactically correct. Not found a way to achieve this yet.

@alister
The new(ish) thing is the Shell reacting to unquoted square brackets which is why we need to quote them in a tr.

Code:

ls ?
A  B  C

ls [A-C]
A  B  C

echo [A-C]
A B C

Edit: Noted that @alister has found a way of generating the anomoly. See subsequent posts.

It's trivial, in certain environments. I haven't used linux in a long while, but I still have a 2006 Debian install on an old laptop. It uses the en_US.UTF-8 locale by default. Here's one way to replicate OP's issue (command output bolded):

Code:

$ locale | grep COLLATE
LC_COLLATE="en_US.UTF-8"
$ mkdir test
$ cd test
$ set -x
$ echo NEE | tr [A-Z] [a-z]
+ tr '[A-Z]' '[a-z]'
+ echo NEE
nee
$ touch F
+ touch F
$ echo NEE | tr [A-Z] [a-z]
+ tr F F
+ echo NEE
NEE

Note that both [A-Z] and [a-z] matched F, because the locale's collation sequence interleaves uppercase and lowercase letters, including F in both sh glob patterns.

Since many implementations will simply ignore excess characters in the second string, they can be used to reproduce the error even when using a locale which does not interleave upper and lower case letters. Continuing where we left off, in the same working directory and environment:

Code:

$ LC_COLLATE=C
+ LC_COLLATE=C
$ echo NEE | tr [A-Z] [a-z]
+ tr F '[a-z]'
+ echo NEE
NEE

If your locale's collation isn't interleaved and if your tr implementation will not accept a longer second set of characters, then you will not be able to reproduce either result.

Regards,
Alister

It has nothing to do with tr. It'd happen anywhere you use an unquoted [a-z].

Code:

$ touch a b c d e f g h i j k l m n o p q r s t u v w x y z
$ echo [a-z]
a b c d e f g h i j k l m n o p q r s t u v w x y z
$