Hi,
I have a non-ascii character (Ŵ), which can be represented in UTF-8 encoding as equivalent hex value (\xC5B4). Is there a function in unix to convert this hex value back to display the charcter ?

If your locale is set up correctly any number of utilities can display this character correctly. For example if your shell is ksh93 version s or better, printf "\xC5B4" will output the expected character.

Hi fpmurphy, thanks for the response.
can u pls elaborate on how and what locale do is set ? i am operating on solaris 5.8 OS.

also this is a part of the bigger problem. Actually i am transmitting the above character through a email message. so i extract this message from the mail server in unix, and decode it. What happens here is that the character is decoded as (Å´), which is actually 0xC5B4 annoted in ascii ( 0xC5 = Å, 0xB4 = ´).

So i want to take these (Å´) and convert to (Ŵ) directly or via their hex value (\xC5B4).

That means the decoding process treats the message as ISO-8859-1 (or ASCII) rather than UTF-8. There is no "conversion" going on here. It's simply the decoding process fails based on a wrong assumption of encoding.

Have you tried to investigate if anything is wrong that causes the message not to be interpreted as UTF-8? For instance, did you check the encoding in the mail header, was it erroneously specified as anything other than UTF-8? And you may try with other mail with UTF-8 and see if that is an issue with a particular mail (sometimes a misconfigured mail user agent is culprit) or a bigger issue. Try switching mail clients and see if you can always reproduce that.

Hi there,
I am actually using perl to retrieve message from the mailbox. the perl module for encoding/decoding (MIME-Base64-3.07 > MIME::Base64) is the one to be used, but while decoding it does decode to ascii/iso-8859-1 (while the mail header correctly shows the encoding as UTF-8).

In this case, if i want to convert this data to utf-8 back (as detailed above) , is there a command/way to do it in unix ?

I am not too sure about MIME::Base64 as I have not used it before. However, base64 itself is encoding-agnostic, that is, it encodes/decodes without regard to whatever encoding the original message is, because it is not only used to encode textual data, but also images, zip files or just about any binary data you can imagine, that do not have the notion of an "encoding" at all. So what Base64 sees and acts on, is just a bytestream. it doesn't really care what is inside.

So, for a text message:

            encoding      Base64 Encoding
Text content ---> bytestream ---> Base64-encoded message

               Base64 decoding      Decoding
Base64-encoded message ---> bytestream ---> Text content

Ok, in case you feel bewildered by that manpage (you probably will!), let me give you a series of examples to give you a general idea of some of the most important things you need to know.

As I'm Chinese, I'll use Chinese in the examples. All the code are in UTF-8.

Expected environment: a UTF-8 terminal with proper fonts to render Unicode text.

Test 1 - Let's start with this

my $str = "你好吗？";

print("$str\n");
printf("Length: %d\n", length($str));

use utf8;

my $str = "你好吗？";

print("$str\n");
printf("Length: %d\n", length($str));

use utf8;
binmode(*STDOUT, ":utf8");

my $str = "你好吗？";

print("$str\n");
printf("Length: %d\n", length($str));

use Encode;
binmode(*STDOUT, ":utf8");

my $str = "你好吗？";
$str = decode('utf8', $str);

print("$str\n");
printf("Length: %d\n", length($str));