Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

ksh hidden characters in variables

Shell Programming and Scripting


Tags
ksh, rhel 6

Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by user052009
user052009's Unix or Linux Image
user052009 user052009 is offline
Registered User
 
Join Date: Nov 2016
Last Activity: 21 November 2017, 11:49 AM EST
Posts: 57
Thanks: 25
Thanked 1 Time in 1 Post
ksh hidden characters in variables

Hi. I'm getting the following hidden characters
Code:
\u[2013]

at the start of a string after I pass in variables from the command line. I only noticed this when I set -x in my KSH script. Can anybody tell me how this happens and how to remove them?
Many thanks.


Code:
+ STR=$'\u[2013]username testuser1'
+ print $'STR Variable: \u[2013]username testuser1'
STR Variable: username testuser1

Sponsored Links
    #2  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by user052009
Scrutinizer's Unix or Linux Image
Scrutinizer Scrutinizer is online now Forum Staff  
Moderator
 
Join Date: Nov 2008
Last Activity: 13 December 2017, 9:07 PM EST
Location: Amsterdam
Posts: 11,650
Thanks: 518
Thanked 3,385 Times in 2,984 Posts
I think it may be an UTF-16 character.

What is the output of the following command on your command line:

Code:
locale

Sponsored Links
    #3  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by user052009
user052009's Unix or Linux Image
user052009 user052009 is offline
Registered User
 
Join Date: Nov 2016
Last Activity: 21 November 2017, 11:49 AM EST
Posts: 57
Thanks: 25
Thanked 1 Time in 1 Post
Looks like it's UTF-8. Any idea how can I get rid of this hidden character?


Code:
+ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

    #4  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by user052009
RudiC's Unix or Linux Image
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 13 December 2017, 4:17 PM EST
Location: Aachen, Germany
Posts: 11,797
Thanks: 339
Thanked 3,659 Times in 3,357 Posts
How did you call the script? What exactly is in the script?
Did you mayhap use a strange dash character?



Code:
U+2013 EN DASH

System	Representation
Nº	8211
UTF-8	E2 80 93
UTF-16	20 13                  
UTF-32	00 00 20 13
URL-Quoted 	%E2%80%93
HTML-Escape	–


Last edited by RudiC; 3 Weeks Ago at 06:19 AM..
Sponsored Links
    #5  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by user052009
user052009's Unix or Linux Image
user052009 user052009 is offline
Registered User
 
Join Date: Nov 2016
Last Activity: 21 November 2017, 11:49 AM EST
Posts: 57
Thanks: 25
Thanked 1 Time in 1 Post
Yes indeed there is some old code preceding this that reads in the params from the command line:


Code:
while [ -n "$1" ]; do
        case $1 in
                -username)       if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { echo "missing value for '$1' (seen '$2')"; usage; exit 1; } else { shift; USERNAME=$1;   } fi ;;
                -surname)    if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { echo "missing value for '$1' (seen '$2')"; usage; exit 1; } else { shift; SURNAME=$1;     } fi ;;
                -address)    if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { ADDRESS=NO; }    else { shift; ADDRESS=$1;     } fi ;;
                -startdate)  if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { STARTDATE=NO; UNTIL=NO;}   else { shift; STARTDATE=$1; UNTIL=$1;  } fi ;;
                -req)        if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { REQ=Y; }    else { shift; REQ=$1;    } fi ;;
                -opt)
                        if [ -n "$2" ]; then
                                shift; OPT=$*
                                break
                        else
                                OPT=NOTSET
                        fi
                        ;;
                *)             echo "Invalid argument '$1'"; exit 1;;
        esac
        shift
done

I just want to parse the OPT variable after this.

---------- Post updated at 09:44 PM ---------- Previous update was at 09:53 AM ----------

These hidden characters are added on the line:

Code:
shift; OPT=$*


Last edited by user052009; 3 Weeks Ago at 11:30 PM..
Sponsored Links
    #6  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by user052009
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is online now Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 13 December 2017, 8:59 PM EST
Location: San Jose, CA, USA
Posts: 10,774
Thanks: 590
Thanked 3,771 Times in 3,219 Posts
Quote:
Originally Posted by user052009 View Post
Yes indeed there is some old code preceding this that reads in the params from the command line:


Code:
while [ -n "$1" ]; do
        case $1 in
                -username)       if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { echo "missing value for '$1' (seen '$2')"; usage; exit 1; } else { shift; USERNAME=$1;   } fi ;;
                -surname)    if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { echo "missing value for '$1' (seen '$2')"; usage; exit 1; } else { shift; SURNAME=$1;     } fi ;;
                -address)    if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { ADDRESS=NO; }    else { shift; ADDRESS=$1;     } fi ;;
                -startdate)  if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { STARTDATE=NO; UNTIL=NO;}   else { shift; STARTDATE=$1; UNTIL=$1;  } fi ;;
                -req)        if [ "`echo $2 | grep -e '^-[a-z]'`" ]; then { REQ=Y; }    else { shift; REQ=$1;    } fi ;;
                -opt)
                        if [ -n "$2" ]; then
                                shift; OPT=$*
                                break
                        else
                                OPT=NOTSET
                        fi
                        ;;
                *)             echo "Invalid argument '$1'"; exit 1;;
        esac
        shift
done

I just want to parse the OPT variable after this.

---------- Post updated at 09:44 PM ---------- Previous update was at 09:53 AM ----------

These hidden characters are added on the line:

Code:
shift; OPT=$*

If the characters are stored in OPT by the command OPT=$*, then those characters were typed on the command line when your script was invoked.

One would not expect this if your script was invoked by typing commands into a shell.

One would not expect this if your script was invoked by a shell script edited with ed, ex, vi or another common UNIX system text editor.

One would expect this if your script was invoked by a shell script edited with a text editor designed to produce pretty-printed text (such as Microsoft word or Apple pages).

Last edited by Don Cragun; 3 Weeks Ago at 11:55 PM.. Reason: Add emphasis.
Sponsored Links
    #7  
Old Unix and Linux 3 Weeks Ago   -   Original Discussion by user052009
RudiC's Unix or Linux Image
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 13 December 2017, 4:17 PM EST
Location: Aachen, Germany
Posts: 11,797
Thanks: 339
Thanked 3,659 Times in 3,357 Posts
I second Don Cragun in his third expectation; that's why I asked how you called the script (which unfortunately you did not answer). It looks like your parent script scrambles character (set)s as the case statement evaluates and recognizes -OPT correctly but then obviously fails when providing the option values themselves.
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Hidden characters when pasting in vi/vim dampio UNIX for Beginners Questions & Answers 3 12-02-2016 01:59 PM
How to see hidden characters..... Bashingaway Shell Programming and Scripting 8 11-03-2010 12:49 PM
hidden Characters Khoomfire UNIX for Advanced & Expert Users 5 07-23-2008 10:11 AM
Hidden Characters Khoomfire UNIX for Advanced & Expert Users 6 07-06-2007 05:40 AM
missing hidden characters SUSANR9999 UNIX for Dummies Questions & Answers 2 11-09-2006 04:30 PM



All times are GMT -4. The time now is 10:13 PM.