ksh hidden characters in variables


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting ksh hidden characters in variables
# 1  
Old 11-16-2017
ksh hidden characters in variables

Hi. I'm getting the following hidden characters
Code:
\u[2013]

at the start of a string after I pass in variables from the command line. I only noticed this when I set -x in my KSH script. Can anybody tell me how this happens and how to remove them?
Many thanks.

Code:
+ STR=$'\u[2013]username testuser1'
+ print $'STR Variable: \u[2013]username testuser1'
STR Variable: username testuser1

# 2  
Old 11-17-2017
I think it may be an UTF-16 character.

What is the output of the following command on your command line:
Code:
locale

# 3  
Old 11-17-2017
Looks like it's UTF-8. Any idea how can I get rid of this hidden character?

Code:
+ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

# 4  
Old 11-17-2017
How did you call the script? What exactly is in the script?
Did you mayhap use a strange dash character?


Code:
U+2013 EN DASH

System	Representation
Nš	8211
UTF-8	E2 80 93
UTF-16	20 13                  
UTF-32	00 00 20 13
URL-Quoted 	%E2%80%93
HTML-Escape	–


Last edited by RudiC; 11-17-2017 at 06:19 AM..
# 5  
Old 11-17-2017
Yes indeed there is some old code preceding this that reads in the params from the command line:

Code:
while [ -n "$1" ]; do
        case $1 in
                -username)       if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { echo "missing value for '$1' (seen '$2')"; usage; exit 1; } else { shift; USERNAME=$1;   } fi ;;
                -surname)    if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { echo "missing value for '$1' (seen '$2')"; usage; exit 1; } else { shift; SURNAME=$1;     } fi ;;
                -address)    if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { ADDRESS=NO; }    else { shift; ADDRESS=$1;     } fi ;;
                -startdate)  if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { STARTDATE=NO; UNTIL=NO;}   else { shift; STARTDATE=$1; UNTIL=$1;  } fi ;;
                -req)        if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { REQ=Y; }    else { shift; REQ=$1;    } fi ;;
                -opt)
                        if [ -n "$2" ]; then
                                shift; OPT=$*
                                break
                        else
                                OPT=NOTSET
                        fi
                        ;;
                *)             echo "Invalid argument '$1'"; exit 1;;
        esac
        shift
done

I just want to parse the OPT variable after this.

---------- Post updated at 09:44 PM ---------- Previous update was at 09:53 AM ----------

These hidden characters are added on the line:
Code:
shift; OPT=$*


Last edited by user052009; 11-17-2017 at 11:30 PM..
# 6  
Old 11-17-2017
Quote:
Originally Posted by user052009
Yes indeed there is some old code preceding this that reads in the params from the command line:

Code:
while [ -n "$1" ]; do
        case $1 in
                -username)       if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { echo "missing value for '$1' (seen '$2')"; usage; exit 1; } else { shift; USERNAME=$1;   } fi ;;
                -surname)    if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { echo "missing value for '$1' (seen '$2')"; usage; exit 1; } else { shift; SURNAME=$1;     } fi ;;
                -address)    if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { ADDRESS=NO; }    else { shift; ADDRESS=$1;     } fi ;;
                -startdate)  if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { STARTDATE=NO; UNTIL=NO;}   else { shift; STARTDATE=$1; UNTIL=$1;  } fi ;;
                -req)        if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { REQ=Y; }    else { shift; REQ=$1;    } fi ;;
                -opt)
                        if [ -n "$2" ]; then
                                shift; OPT=$*
                                break
                        else
                                OPT=NOTSET
                        fi
                        ;;
                *)             echo "Invalid argument '$1'"; exit 1;;
        esac
        shift
done

I just want to parse the OPT variable after this.

---------- Post updated at 09:44 PM ---------- Previous update was at 09:53 AM ----------

These hidden characters are added on the line:
Code:
shift; OPT=$*

If the characters are stored in OPT by the command OPT=$*, then those characters were typed on the command line when your script was invoked.

One would not expect this if your script was invoked by typing commands into a shell.

One would not expect this if your script was invoked by a shell script edited with ed, ex, vi or another common UNIX system text editor.

One would expect this if your script was invoked by a shell script edited with a text editor designed to produce pretty-printed text (such as Microsoft word or Apple pages).

Last edited by Don Cragun; 11-17-2017 at 11:55 PM.. Reason: Add emphasis.
# 7  
Old 11-18-2017
I second Don Cragun in his third expectation; that's why I asked how you called the script (which unfortunately you did not answer). It looks like your parent script scrambles character (set)s as the case statement evaluates and recognizes -OPT correctly but then obviously fails when providing the option values themselves.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Rsync - how to copy hidden folder or hidden files when using full path

Hello. I use this command : rsync -av --include=".*" --dry-run "$A_FULL_PATH_S" "$A_FULL_PATH_D"The data comes from the output of a find command. And no full source directories are in use, only some files. Source example... (2 Replies)
Discussion started by: jcdole
2 Replies

2. UNIX for Beginners Questions & Answers

Hidden characters when pasting in vi/vim

Hello everyone. When I copy some configuration settings string from MS_Word to putty from my personal pc to a remote machine, it appears that I copy some hidden symbols, which at first, cannot be seen and appear as hidden. Some java programs did not start, and after investigation I found that: ... (3 Replies)
Discussion started by: dampio
3 Replies

3. Shell Programming and Scripting

Hidden Characters in Regular Expression Matching Perl - Perl Newbie

I am completely new to perl programming. My father is helping me learn said programming language. However, I am stuck on one of the assignments he has given me, and I can't find very much help with it via google, either because I have a tiny attention span, or because I can be very very dense. ... (4 Replies)
Discussion started by: kittyluva2
4 Replies

4. Shell Programming and Scripting

How to see hidden characters.....

I know that cat -v will show me hidden characters in a file.... I for some reason seem to think that there's a bash command that will show me hidden characters in a variable in a script? Or am I just imagining it? Thanks in advance (8 Replies)
Discussion started by: Bashingaway
8 Replies

5. Shell Programming and Scripting

hidden characters added to commands

Hello everybody. Im really new in shell scripting. Im working with RedHat 4. I have begin to do some scripts to test the posibilitys but Im fancing a disturbing problem. some times the lines that I write add the return character or end-of-file ascii character to the command or argument tha... (2 Replies)
Discussion started by: josebenitez
2 Replies

6. Shell Programming and Scripting

Hidden control characters in a Unix Text File!

Can anyone seem to know how to find out whether a UNIX text file has 'hidden' control characters? Can I view them using 'vi' by some command line options? If there are control characters in a text file which are invisible/hidden.. then how do I get rid of them? Your intelletual answers are... (6 Replies)
Discussion started by: kewl_guy
6 Replies

7. UNIX for Advanced & Expert Users

hidden Characters

Hello All, I'm trying to parse through a file and display all the hidden characters, including all carriage and line returns. I usually use cat -v, but this doesn't display the carriage and line returns. Does anyone know how this can be done? Thanks Khoom (5 Replies)
Discussion started by: Khoomfire
5 Replies

8. UNIX for Advanced & Expert Users

Hidden Characters

Hello all, I have two files, that I suspect may contain hidden characters (EG, three spaces instead of a tab). Does anyone know of any tool that can display this (I have tried using diff, but I'm not quite sure it would do the job) (6 Replies)
Discussion started by: Khoomfire
6 Replies

9. UNIX for Dummies Questions & Answers

missing hidden characters

I made a small change to a UNIX script. When I tried to run it I received the following message: /bin/ksh: ^M: not found /bin/ksh: ^M: not found /bin/ksh: ^M: not found As ^M is a non printing character, I don't know how to discover where it is missing. How can I correct thiis... (2 Replies)
Discussion started by: SUSANR9999
2 Replies
Login or Register to Ask a Question