Possible ksh93 Bug Expanding Variables?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Possible ksh93 Bug Expanding Variables?
# 1  
Old 07-03-2017
Possible ksh93 Bug Expanding Variables?

My OS is Linux (kernel 4.08.something) and AIX (7100-04-01-1543), the used ksh versions are:

ksh88: Version M-11/16/88f (AIX)
ksh93: Version M 93t+ 2009-05-01 (AIX), Version M 93u (Linux)

When writing a parser for stanza files in ksh i encountered a rather strange behavior. Here is a stripped down version of the parsing loop (i left out the actual parsing for clarity if anyone is interested i can post it), i marked the two critical lines in bold:

Code:
#! /bin/ksh

typeset chLine=""
typeset chChar=""

while read chLine ; do
     print - "-- Begin Line: $chLine"
     while [ -n "$chLine" ] ; do
          chChar="${chLine%${chLine#?}}"
          chLine="${chLine#?}"

          print - "\n  Line: \"$chLine\""
          print - "  Char: \"$chChar\""

     done
done < /iput/file

exit 0

Here is the output of this with the sample file (1 line) containing item=value, which works as expected:

Code:
# ./parsetest.sh  
-- Begin Line: item=value

  Line: "tem=value"
  Char: "i"

  Line: "em=value"
  Char: "t"

  Line: "m=value"
  Char: "e"

  Line: "=value"
  Char: "m"

  Line: "value"
  Char: "="

  Line: "alue"
  Char: "v"

  Line: "lue"
  Char: "a"

  Line: "ue"
  Char: "l"

  Line: "e"
  Char: "u"

  Line: ""
  Char: "e"

Now, because i wanted to have comments in my stanza files (which the parser should filter out), but needed to make escaped comment chars possible i tried with the line item=val\\#ue. Here is the output running above script with with ksh88, which is as expected:
Code:
-- Begin Line: item=val\#ue

  Line: "tem=val\#ue"
  Char: "i"

  Line: "em=val\#ue"
  Char: "t"

  Line: "m=val\#ue"
  Char: "e"

  Line: "=val\#ue"
  Char: "m"

  Line: "val\#ue"
  Char: "="

  Line: "al\#ue"
  Char: "v"

  Line: "l\#ue"
  Char: "a"

  Line: "\#ue"
  Char: "l"

  Line: "#ue"
  Char: "\"

  Line: "ue"
  Char: "#"

  Line: "e"
  Char: "u"

  Line: ""
  Char: "e"

But - and this is where it gets weird - if the script is run with ksh93, the characterwise chopping off from the main string stops to work correctly:
Code:
# ./parsetest.sh     
-- Begin Line: item=val\#ue

  Line: "tem=val\#ue"
  Char: "item=val\#ue"

  Line: "em=val\#ue"
  Char: "tem=val\#ue"

  Line: "m=val\#ue"
  Char: "em=val\#ue"

  Line: "=val\#ue"
  Char: "m=val\#ue"

  Line: "val\#ue"
  Char: "=val\#ue"

  Line: "al\#ue"
  Char: "val\#ue"

  Line: "l\#ue"
  Char: "al\#ue"

  Line: "\#ue"
  Char: "l\"

  Line: "#ue"
  Char: "\"

  Line: "ue"
  Char: "#"

  Line: "e"
  Char: "u"

  Line: ""
  Char: "e"

Notice, that as long as the escape char is present in the string the variable expansion in the two marked lines seems not to work correctly.

Does anyone have an explanation for this or have i just encountered a bug?

bakunin
# 2  
Old 07-03-2017
I'm not explaining, but I get the same result in bash and dash...

Regards.
# 3  
Old 07-03-2017
You are hoping the backslash in the expansion of ${chLine#?} will be treated as a literal backslash character. And, in the expansion of ${chLine} it is. But, in word in ${chLine%word} the backslash is an escape character. Since \# is treated as an escaped # in the pattern instead of the two character literal \#, there is no match and the string isn't removed from the expansion with the removal of a matching smallest suffix pattern.

This is one of the changes that was made to ksh88 behavior (that is handled differently in ksh93) while the POSIX shell standard was being developed.

I think this explains the difference you're seeing, but unless your script tests which version of ksh you're using and uses different code for the two cases, you may have trouble finding a common variable expansion that will get you what you want in both versions of the shell. Unfortunately, you can't use ${chLine:2} in ksh88 (but it gives you what you want in ksh93).
This User Gave Thanks to Don Cragun For This Post:
# 4  
Old 07-04-2017
Maybe you'd like to try the following which should work with ksh88 and any POSIX conforming shell (including ksh93):
Code:
#! /bin/ksh

typeset chLine=""
typeset chChar=""
typeset pattern='??????????'

while read -r chLine
do	print - "-- Begin Line: $chLine"
	while [ ${#chLine} -gt ${#pattern} ]
	do	pattern="$pattern$pattern"
	done
	while [ -n "$chLine" ]
	do	chChar="${chLine%$(printf '%*.*s' $((${#chLine} - 1)) \
		    $((${#chLine} - 1)) "$pattern")}"
		chLine="${chLine#?}"

		print - "\n  Line: \"$chLine\""
		print - "  Char: \"$chChar\""
	done
done < /iput/file

exit 0

If /iput/file contains:
Code:
item=value
item=val\#ue
[{(This is line 3*)}]\.

the above code should produce the output:
Code:
-- Begin Line: item=value

  Line: "tem=value"
  Char: "i"

  Line: "em=value"
  Char: "t"

  Line: "m=value"
  Char: "e"

  Line: "=value"
  Char: "m"

  Line: "value"
  Char: "="

  Line: "alue"
  Char: "v"

  Line: "lue"
  Char: "a"

  Line: "ue"
  Char: "l"

  Line: "e"
  Char: "u"

  Line: ""
  Char: "e"
-- Begin Line: item=val\#ue

  Line: "tem=val\#ue"
  Char: "i"

  Line: "em=val\#ue"
  Char: "t"

  Line: "m=val\#ue"
  Char: "e"

  Line: "=val\#ue"
  Char: "m"

  Line: "val\#ue"
  Char: "="

  Line: "al\#ue"
  Char: "v"

  Line: "l\#ue"
  Char: "a"

  Line: "\#ue"
  Char: "l"

  Line: "#ue"
  Char: "\"

  Line: "ue"
  Char: "#"

  Line: "e"
  Char: "u"

  Line: ""
  Char: "e"
-- Begin Line: [{(This is line 3*)}]\.

  Line: "{(This is line 3*)}]\."
  Char: "["

  Line: "(This is line 3*)}]\."
  Char: "{"

  Line: "This is line 3*)}]\."
  Char: "("

  Line: "his is line 3*)}]\."
  Char: "T"

  Line: "is is line 3*)}]\."
  Char: "h"

  Line: "s is line 3*)}]\."
  Char: "i"

  Line: " is line 3*)}]\."
  Char: "s"

  Line: "is line 3*)}]\."
  Char: " "

  Line: "s line 3*)}]\."
  Char: "i"

  Line: " line 3*)}]\."
  Char: "s"

  Line: "line 3*)}]\."
  Char: " "

  Line: "ine 3*)}]\."
  Char: "l"

  Line: "ne 3*)}]\."
  Char: "i"

  Line: "e 3*)}]\."
  Char: "n"

  Line: " 3*)}]\."
  Char: "e"

  Line: "3*)}]\."
  Char: " "

  Line: "*)}]\."
  Char: "3"

  Line: ")}]\."
  Char: "*"

  Line: "}]\."
  Char: ")"

  Line: "]\."
  Char: "}"

  Line: "\."
  Char: "]"

  Line: "."
  Char: "\"

  Line: ""
  Char: "."

This User Gave Thanks to Don Cragun For This Post:
# 5  
Old 07-04-2017
Hats off to your debugging skills, Don!

Quote:
Originally Posted by Don Cragun
You are hoping the backslash in the expansion of ${chLine#?} will be treated as a literal backslash character. And, in the expansion of ${chLine} it is. But, in word in ${chLine%word} the backslash is an escape character.
After this i finally understood what the problem was in first place. Thank you for enlightening me.

Quote:
Originally Posted by Don Cragun
Maybe you'd like to try the following which should work with ksh88 and any POSIX conforming shell (including ksh93):
Thank you again for providing even a solution along with the explanation. I was already (reluctantly) dusting off my trusted old C compiler to write the parser there. I am indebted.

bakunin

Last edited by bakunin; 07-04-2017 at 03:41 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

[BASH] eval command not expanding variables as expected.

Hi Guys, I wrote a collection of bash functions years ago and now need to use them again but I'm getting some error messages when eval tries to expand the variables names. I recollect that I used the shopt command to set one of the options but I can't quite remember the command that I... (8 Replies)
Discussion started by: ASGR
8 Replies

2. AIX

Ksh93/AIX compatibility

Hi everyone ! Im trying to know from wich version of AIX KSH93 is available ? Internet tell me 6.x and 7.x AIX are available, bue what about 5.x ? Is KSH93 available on AIX 5.x ? Is it the same way to manipulate variables as KSH93 on 7.x ? Thanks for your support and have a nice day ! (2 Replies)
Discussion started by: majinfrede
2 Replies

3. UNIX for Advanced & Expert Users

Ksh93 on Linux compatible with ksh93 on AIX

Hi Experts, I have several shell scripts that have been developed on a Linux box for korn ksh93. If we want to run this software on an AIX 6.1 box that runs ksh88 by default can we just change the she-bang line to reference /bin/ksh93 which ships with AIX as its "enhanced shell" to ensure... (6 Replies)
Discussion started by: Keith Turley
6 Replies

4. Shell Programming and Scripting

awk issue expanding variables in ksh script

Hi Guys, I have an issue with awk and variables. I have trawled the internet and forums but can't seem to get the exactt syntax I need. I have tried using awk -v and all sorts of variations but I have hit a brick wall. I have spent a full day on this and am just going round in circles. ... (3 Replies)
Discussion started by: gazza-o
3 Replies

5. UNIX for Dummies Questions & Answers

Expanding variables with Ed Bash 3.2.33

Hi, The following code finds the line containing fruits in test.txt and replaces instances of apple with banana. ed -s test.txt <<< $'/fruits/s/apple/banana/g\nw' What I want to do is put variables in the place of fruits, apple and banana. I have tried replacing ' with " to get... (2 Replies)
Discussion started by: de_la_espada
2 Replies

6. Shell Programming and Scripting

Performance degradation with KSH93

Hi, I have a script that calls an external program to perform some calculations and then I read with "grep" and "sed" values from the output files. I've noticed that performance of KSH93 degrades with every iteration. The output files are all the same size, so I don't understand why after the... (2 Replies)
Discussion started by: i.f.schulz
2 Replies

7. Shell Programming and Scripting

ksh88 or ksh93

Hi all! Does anybody know how can I check if any UNIX installation has implemented ksh88 or ksh93? Thanks in advance. Néstor. (3 Replies)
Discussion started by: Nestor
3 Replies

8. Shell Programming and Scripting

ksh93 deprecation...

Any means of running ksh93 in a ksh88-mode? Might sound odd, but I want/need to restrict U/Win-developed scripts to correspond to the ksh88 version on my Solaris environment(s). Thanks. (2 Replies)
Discussion started by: curleb
2 Replies

9. UNIX for Advanced & Expert Users

Expanding Variables in User Input

If have var='$variable' how can I expand $variable. I have tried many thing like duble quotes/braces etc, but nothing worked. I need the solution ASAP. (2 Replies)
Discussion started by: Bsk
2 Replies
Login or Register to Ask a Question