Shell Programming and Scripting

BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

ksh hidden characters in variables


👤 Login to reply

 
Thread Tools Search this Thread Display Modes
    #1  
Old 11-16-2017
user052009 user052009 is offline
Registered User
 
Join Date: Nov 2016
Last Activity: 12 January 2018, 4:56 AM EST
Posts: 57
Thanks: 25
Thanked 1 Time in 1 Post
ksh hidden characters in variables

Hi. I'm getting the following hidden characters
Code:
\u[2013]

at the start of a string after I pass in variables from the command line. I only noticed this when I set -x in my KSH script. Can anybody tell me how this happens and how to remove them?
Many thanks.

Code:
+ STR=$'\u[2013]username testuser1'
+ print $'STR Variable: \u[2013]username testuser1'
STR Variable: username testuser1

Sponsored Links
    #2  
Old 11-17-2017
Scrutinizer's Unix or Linux Image
Scrutinizer Scrutinizer is offline Forum Staff  
Moderator
 
Join Date: Nov 2008
Last Activity: 19 July 2018, 1:50 AM EDT
Location: Amsterdam
Posts: 11,853
Thanks: 546
Thanked 3,469 Times in 3,057 Posts
I think it may be an UTF-16 character.

What is the output of the following command on your command line:
Code:
locale

Sponsored Links
    #3  
Old 11-17-2017
user052009 user052009 is offline
Registered User
 
Join Date: Nov 2016
Last Activity: 12 January 2018, 4:56 AM EST
Posts: 57
Thanks: 25
Thanked 1 Time in 1 Post
Looks like it's UTF-8. Any idea how can I get rid of this hidden character?

Code:
+ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

    #4  
Old 11-17-2017
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 19 July 2018, 5:31 PM EDT
Location: Aachen, Germany
Posts: 13,077
Thanks: 450
Thanked 4,016 Times in 3,693 Posts
How did you call the script? What exactly is in the script?
Did you mayhap use a strange dash character?


Code:
U+2013 EN DASH

System	Representation
Nº	8211
UTF-8	E2 80 93
UTF-16	20 13                  
UTF-32	00 00 20 13
URL-Quoted 	%E2%80%93
HTML-Escape	–


Last edited by RudiC; 11-17-2017 at 05:19 AM..
Sponsored Links
    #5  
Old 11-17-2017
user052009 user052009 is offline
Registered User
 
Join Date: Nov 2016
Last Activity: 12 January 2018, 4:56 AM EST
Posts: 57
Thanks: 25
Thanked 1 Time in 1 Post
Yes indeed there is some old code preceding this that reads in the params from the command line:

Code:
while [ -n "$1" ]; do
        case $1 in
                -username)       if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { echo "missing value for '$1' (seen '$2')"; usage; exit 1; } else { shift; USERNAME=$1;   } fi ;;
                -surname)    if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { echo "missing value for '$1' (seen '$2')"; usage; exit 1; } else { shift; SURNAME=$1;     } fi ;;
                -address)    if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { ADDRESS=NO; }    else { shift; ADDRESS=$1;     } fi ;;
                -startdate)  if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { STARTDATE=NO; UNTIL=NO;}   else { shift; STARTDATE=$1; UNTIL=$1;  } fi ;;
                -req)        if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { REQ=Y; }    else { shift; REQ=$1;    } fi ;;
                -opt)
                        if [ -n "$2" ]; then
                                shift; OPT=$*
                                break
                        else
                                OPT=NOTSET
                        fi
                        ;;
                *)             echo "Invalid argument '$1'"; exit 1;;
        esac
        shift
done

I just want to parse the OPT variable after this.

---------- Post updated at 09:44 PM ---------- Previous update was at 09:53 AM ----------

These hidden characters are added on the line:
Code:
shift; OPT=$*


Last edited by user052009; 11-17-2017 at 10:30 PM..
Sponsored Links
    #6  
Old 11-17-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 19 July 2018, 5:40 PM EDT
Location: San Jose, CA, USA
Posts: 11,408
Thanks: 651
Thanked 3,970 Times in 3,393 Posts
Quote:
Originally Posted by user052009 View Post
Yes indeed there is some old code preceding this that reads in the params from the command line:

Code:
while [ -n "$1" ]; do
        case $1 in
                -username)       if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { echo "missing value for '$1' (seen '$2')"; usage; exit 1; } else { shift; USERNAME=$1;   } fi ;;
                -surname)    if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { echo "missing value for '$1' (seen '$2')"; usage; exit 1; } else { shift; SURNAME=$1;     } fi ;;
                -address)    if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { ADDRESS=NO; }    else { shift; ADDRESS=$1;     } fi ;;
                -startdate)  if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { STARTDATE=NO; UNTIL=NO;}   else { shift; STARTDATE=$1; UNTIL=$1;  } fi ;;
                -req)        if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { REQ=Y; }    else { shift; REQ=$1;    } fi ;;
                -opt)
                        if [ -n "$2" ]; then
                                shift; OPT=$*
                                break
                        else
                                OPT=NOTSET
                        fi
                        ;;
                *)             echo "Invalid argument '$1'"; exit 1;;
        esac
        shift
done

I just want to parse the OPT variable after this.

---------- Post updated at 09:44 PM ---------- Previous update was at 09:53 AM ----------

These hidden characters are added on the line:
Code:
shift; OPT=$*

If the characters are stored in OPT by the command OPT=$*, then those characters were typed on the command line when your script was invoked.

One would not expect this if your script was invoked by typing commands into a shell.

One would not expect this if your script was invoked by a shell script edited with ed, ex, vi or another common UNIX system text editor.

One would expect this if your script was invoked by a shell script edited with a text editor designed to produce pretty-printed text (such as Microsoft word or Apple pages).

Last edited by Don Cragun; 11-17-2017 at 10:55 PM.. Reason: Add emphasis.
Sponsored Links
    #7  
Old 11-18-2017
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 19 July 2018, 5:31 PM EDT
Location: Aachen, Germany
Posts: 13,077
Thanks: 450
Thanked 4,016 Times in 3,693 Posts
I second Don Cragun in his third expectation; that's why I asked how you called the script (which unfortunately you did not answer). It looks like your parent script scrambles character (set)s as the case statement evaluates and recognizes -OPT correctly but then obviously fails when providing the option values themselves.
Sponsored Links
👤 Login to reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Hidden characters when pasting in vi/vim dampio UNIX for Beginners Questions & Answers 3 12-02-2016 12:59 PM
How to see hidden characters..... Bashingaway Shell Programming and Scripting 8 11-03-2010 11:49 AM
hidden Characters Khoomfire UNIX for Advanced & Expert Users 5 07-23-2008 09:11 AM
Hidden Characters Khoomfire UNIX for Advanced & Expert Users 6 07-06-2007 04:40 AM
missing hidden characters SUSANR9999 UNIX for Dummies Questions & Answers 2 11-09-2006 03:30 PM



All times are GMT -4. The time now is 07:11 PM.

Unix & Linux Forums Content Copyright©1993-2018. All Rights Reserved.
×
UNIX.COM Login
Username:
Password:  
Show Password





Not a Forum Member?
Forgot Password?