ksh hidden characters in variables


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting ksh hidden characters in variables
# 8  
Old 11-18-2017
It's caused when the first character in the options field is a dash. I'm calling the script from the command line. The script was edited using vi. Here's the full output with debug when I use '-options -debug':

Code:
./test.ksh -username user1 -surname user2 -address abc123 -startdate 123 -req 4 -options –debug 
+ print 'INFO - Process arguments ...'
INFO - Process arguments ...
+ [ -n -username ]
+ grep -e '^-[a-z]'
+ echo user1
+ [ '' ]
+ shift
+ SourceTarget=user1
+ shift
+ [ -n -surname ]
+ grep -e '^-[a-z]'
+ echo user2
+ [ '' ]
+ shift
+ DestTarget=user2
+ shift
+ [ -n -address ]
+ grep -e '^-[a-z]'
+ echo abc123
+ [ '' ]
+ shift
+ ADDRESS=abc123
+ shift
+ [ -n -startdate ]
+ grep -e '^-[a-z]'
+ echo 123
+ [ '' ]
+ shift
+ STARTDATE=123
+ UNTIL=123
+ shift
+ [ -n -req ]
+ grep -e '^-[a-z]'
+ echo 4
+ [ '' ]
+ shift
+ REQ=4
+ shift
+ [ -n -options ]
+ [ -n $'\u[2013]debug' ]
+ shift
+ OPT=$'\u[2013]debug'
+ break
+ print 'INFO - Processed inputted arguments...'
INFO - Processed inputted arguments...
+ STR=$'\u[2013]debug'
+ print $'STR Options field: \u[2013]debug'
STR Options field: –debug
+ exit 0

And now if I run the same command but remove the dash from before 'debug' in the options field:
Code:
./test.ksh -username user1 -surname user2 -address abc123 -startdate 123 -req 4 -options debug 
+ print 'INFO - Process arguments ...'
INFO - Process arguments ...
+ [ -n -username ]
+ grep -e '^-[a-z]'
+ echo user1
+ [ '' ]
+ shift
+ SourceTarget=user1
+ shift
+ [ -n -surname ]
+ grep -e '^-[a-z]'
+ echo user2
+ [ '' ]
+ shift
+ DestTarget=user2
+ shift
+ [ -n -address ]
+ grep -e '^-[a-z]'
+ echo abc123
+ [ '' ]
+ shift
+ ADDRESS=abc123
+ shift
+ [ -n -startdate ]
+ grep -e '^-[a-z]'
+ echo 123
+ [ '' ]
+ shift
+ STARTDATE=123
+ UNTIL=123
+ shift
+ [ -n -req ]
+ grep -e '^-[a-z]'
+ echo 4
+ [ '' ]
+ shift
+ REQ=4
+ shift
+ [ -n -options ]
+ [ -n debug ]
+ shift
+ OPT=debug
+ break
+ print 'INFO - Processed inputted arguments...'
INFO - Processed inputted arguments...
+ STR=debug
+ print 'STR Options field: debug'
STR Options field: debug
+ exit 0


Last edited by user052009; 11-18-2017 at 10:27 PM..
# 9  
Old 11-18-2017
Let us take a close look at the command line you typed into your shell:
Code:
./test.ksh -username user1 -surname user2 -address abc123 -startdate 123 -req 4 -options -debug

If you look closely, you'll see that the character before "username", "surname", "address", "startdate", "req", and "options" is a <hyphen> or <minus-sign>, but the character before "debug" is wider. It is what Unicode calls an <en-dash>. If we feed that line through od and look at it as octal bytes and characters we can easily see the difference:
Code:
echo "./test.ksh -username user1 -surname user2 -address abc123 -startdate 123 -req 4 -options -debug" | od -bc

which produces the output:
Code:
0000000   056 057 164 145 163 164 056 153 163 150 040 055 165 163 145 162
           .   /   t   e   s   t   .   k   s   h       -   u   s   e   r
0000020   156 141 155 145 040 165 163 145 162 061 040 055 163 165 162 156
           n   a   m   e       u   s   e   r   1       -   s   u   r   n
0000040   141 155 145 040 165 163 145 162 062 040 055 141 144 144 162 145
           a   m   e       u   s   e   r   2       -   a   d   d   r   e
0000060   163 163 040 141 142 143 061 062 063 040 055 163 164 141 162 164
           s   s       a   b   c   1   2   3       -   s   t   a   r   t
0000100   144 141 164 145 040 061 062 063 040 055 162 145 161 040 064 040
           d   a   t   e       1   2   3       -   r   e   q       4    
0000120   055 157 160 164 151 157 156 163 040 342 200 223 144 145 142 165
           -   o   p   t   i   o   n   s       -  **  **   d   e   b   u
0000140   147 012                                                        
           g  \n                                                        
0000142

Note that I marked the last hyphen and the en-dash in red in the od output. Each of the hyphens is a single byte with octal value 055 while the en-dash is three bytes with the octal values 342, 200, and 223, respectively.

When you give ksh an en-dash as an input character, it will give it back you you as an en-dash (and not convert it to a hyphen). If you want a the string -debug with a hypen to be assigned to OPTS, give ksh a command line argument containing -debug with a hypen; not -debug with an en-dash.
These 3 Users Gave Thanks to Don Cragun For This Post:
# 10  
Old 11-19-2017
That's brilliant. I hadn't seen the en-dash but it explains everything perfectly. Thank you so much for clearing this up.
# 11  
Old 11-19-2017
There remains the question how you managed to create an en-dash in vi or command line.
# 12  
Old 11-19-2017
A common cause is copy/pasting from a web page or a document where a word processor has replaced two consecutive minus-hyphen characters with a wide single one (short dash).

This is done by default at least with MS Word 2010.
# 13  
Old 11-19-2017
I can only imagine I somehow pressed shift alt (on a mac) along with - when typing that in. It was the only place it occurred. It's great to now have an approach for working this out. Thanks guys.

Last edited by user052009; 11-19-2017 at 07:47 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Rsync - how to copy hidden folder or hidden files when using full path

Hello. I use this command : rsync -av --include=".*" --dry-run "$A_FULL_PATH_S" "$A_FULL_PATH_D"The data comes from the output of a find command. And no full source directories are in use, only some files. Source example... (2 Replies)
Discussion started by: jcdole
2 Replies

2. UNIX for Beginners Questions & Answers

Hidden characters when pasting in vi/vim

Hello everyone. When I copy some configuration settings string from MS_Word to putty from my personal pc to a remote machine, it appears that I copy some hidden symbols, which at first, cannot be seen and appear as hidden. Some java programs did not start, and after investigation I found that: ... (3 Replies)
Discussion started by: dampio
3 Replies

3. Shell Programming and Scripting

Hidden Characters in Regular Expression Matching Perl - Perl Newbie

I am completely new to perl programming. My father is helping me learn said programming language. However, I am stuck on one of the assignments he has given me, and I can't find very much help with it via google, either because I have a tiny attention span, or because I can be very very dense. ... (4 Replies)
Discussion started by: kittyluva2
4 Replies

4. Shell Programming and Scripting

How to see hidden characters.....

I know that cat -v will show me hidden characters in a file.... I for some reason seem to think that there's a bash command that will show me hidden characters in a variable in a script? Or am I just imagining it? Thanks in advance (8 Replies)
Discussion started by: Bashingaway
8 Replies

5. Shell Programming and Scripting

hidden characters added to commands

Hello everybody. Im really new in shell scripting. Im working with RedHat 4. I have begin to do some scripts to test the posibilitys but Im fancing a disturbing problem. some times the lines that I write add the return character or end-of-file ascii character to the command or argument tha... (2 Replies)
Discussion started by: josebenitez
2 Replies

6. Shell Programming and Scripting

Hidden control characters in a Unix Text File!

Can anyone seem to know how to find out whether a UNIX text file has 'hidden' control characters? Can I view them using 'vi' by some command line options? If there are control characters in a text file which are invisible/hidden.. then how do I get rid of them? Your intelletual answers are... (6 Replies)
Discussion started by: kewl_guy
6 Replies

7. UNIX for Advanced & Expert Users

hidden Characters

Hello All, I'm trying to parse through a file and display all the hidden characters, including all carriage and line returns. I usually use cat -v, but this doesn't display the carriage and line returns. Does anyone know how this can be done? Thanks Khoom (5 Replies)
Discussion started by: Khoomfire
5 Replies

8. UNIX for Advanced & Expert Users

Hidden Characters

Hello all, I have two files, that I suspect may contain hidden characters (EG, three spaces instead of a tab). Does anyone know of any tool that can display this (I have tried using diff, but I'm not quite sure it would do the job) (6 Replies)
Discussion started by: Khoomfire
6 Replies

9. UNIX for Dummies Questions & Answers

missing hidden characters

I made a small change to a UNIX script. When I tried to run it I received the following message: /bin/ksh: ^M: not found /bin/ksh: ^M: not found /bin/ksh: ^M: not found As ^M is a non printing character, I don't know how to discover where it is missing. How can I correct thiis... (2 Replies)
Discussion started by: SUSANR9999
2 Replies
Login or Register to Ask a Question