Concerned about C and UNICODE Post: 302084128

Sponsored Content

Top Forums Programming Concerned about C and UNICODE Post 302084128 by tgilbert on Wednesday 9th of August 2006 10:01:19 AM

08-09-2006

Registered User

Concerned about C and UNICODE

Dear experts,

While developping a C UNICODE application under AIX 5.3, I encountered the following problem, and after days of investigations I still could not find any solution.

Please note that the application is full wchar_t based (not utf8) and that I could compile and run it without any problem on SunOS.

I managed to isolate the problem into a simple c program:

Code:

#include <stdio.h>
#include <locale.h>
#include <wchar.h>

  int main()
  {
        wchar_t arab[4] = { 1583, 1575, 1605, 0 };
        wchar_t engl[4] = { 65, 66, 67, 0 };
        wchar_t temp[4] = { 0, 0, 0, 0 };

        printf("\n#1 copy arab into temp");
        wsprintf(temp, "%S", arab);
        printf("\narab bytes : "); for (int i=0; i<4; i++) printf("%d ", (int) arab[i]);
        printf("\ntemp bytes : "); for (int i=0; i<4; i++) printf("%d ", (int) temp[i]);
        printf("\n");

        printf("\n#2 copy engl into temp");
        wsprintf(temp, "%S", engl);
        printf("\nengl bytes : "); for (int i=0; i<4; i++) printf("%d ", (int) engl[i]);
        printf("\ntemp bytes : "); for (int i=0; i<4; i++) printf("%d ", (int) temp[i]);

        printf("\n\n");
        return 0;
  }

Here is the unexpected output:

Code:

#1 copy arab into temp
arab bytes : 1583 1575 1605 0
temp bytes : 32 0 0 0

#2 copy engl into temp
engl bytes : 65 66 67 0
temp bytes : 65 66 67 0

As you can see, the wsprintf call did NOT copy the chars coming from the arabic character set, but did it for the 65, 66, 67 (ABC)... but it copied a single white space (byte 32) instead!

It seems it's related to the installed character sets and to the locales configuration... but then, why using wchar_t strings? This data type and the c functions using it (like wsprintf) are supposed to work whatever the language.

so, my question are:
- What should I do to make this example work ?
- Are the C wchar_t function really character set independent ? (it's the case on Windows and SunOS)
- If necessary, how do I install and use additional character sets ?

Thank you very much for your input about this,

Best regards,

Thomas Gilbert

tgilbert

View Public Profile for tgilbert

Find all posts by tgilbert

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

converting string to unicode

How can I can convert a string in a shell script that looks something like: ]] to unicode equivalent? thanks a lot, webtekie

2. UNIX for Advanced & Expert Users

unicode

Hi, I have some software I need to install on HP-UX 11iv1 64bit but it must not be set up in unicode mode. I know unicode/ ASCII etc. I don't know how to get unix to switch between these. Is there an environment setting for that? I use the Korn shell. Thanks.

3. Programming

How to display unicode characters / unicode string

I have a stream of characters like "\u8BBE\u5907\u7BA1" and i want to display it. I tried following things already without any luck. 1) printf("%s",L("\u8BBE\u5907\u7BA1")); 2) printf("%lc",0x8BBE); 3) setlocale followed by fwide followed by wprintf 4) also changed the local manually...

4. UNIX for Advanced & Expert Users

Unix and Unicode

All, I'm trying to grasp how to use Unicode with/in Unix. I've made progress on some fronts, for example, when uploading files to my server I can use the intermediary language to convert the file to UTF-8. I'm having trouble getting Samba to do this (I'm using "unix charset" in smb.conf);...

5. Programming

unicode problem

on some distributions UTF-32 is the default and i need to change the size of wchar_t to 2 bytes. i tried to compile it with -fwide-exec-charset=UTF-16 but it didn't help. anyone have any ideas? thanks, Akos

6. Programming

Unicode programing in C

im starting to go a little serious with c, woking in a personal project that will read a xml, which might contain Unicode characters (i know it will on my system, which is set to es_AR.UTF-8) im using mxml, and the documentation says it uses utf8 internally (no worries here). so i need to be...

7. Programming

Unicode filenames in C++?

I'm trying to figure out how to support Unicode or atleast an unsigned char in the d_name of struct dirent The problem i'm facing is that I'm checking file names for special characters and obviously the "char d_name" doesn't like it. I'm looping through the directory and getting the file...

8. Shell Programming and Scripting

Help with \u0401 codes ? unicode or something

hello there's some stranges code symbols they looks like this: \u0438 \u0247. unicode i think this code can be viewed by javascript so i need it i need to convert casual characters to this code with perl atm stucked with ord, chr, pack, etc things but they giving other digits

9. Shell Programming and Scripting

Unicode help

is there any way to handle unicode such as ʃʰɐm̆

LEARN ABOUT FREEBSD

wcpncpy

WMEMCHR(3)						   BSD Library Functions Manual 						WMEMCHR(3)

NAME

     wmemchr, wmemcmp, wmemcpy, wmemmove, wmemset, wcpcpy, wcpncpy, wcscasecmp, wcscat, wcschr, wcscmp, wcscpy, wcscspn, wcsdup, wcslcat, wcslcpy,
     wcslen, wcsncasecmp, wcsncat, wcsncmp, wcsncpy, wcsnlen, wcspbrk, wcsrchr, wcsspn, wcsstr -- wide character string manipulation operations

LIBRARY

     Standard C Library (libc, -lc)

SYNOPSIS

     #include <wchar.h>

     wchar_t *
     wmemchr(const wchar_t *s, wchar_t c, size_t n);

     int
     wmemcmp(const wchar_t *s1, const wchar_t *s2, size_t n);

     wchar_t *
     wmemcpy(wchar_t * restrict s1, const wchar_t * restrict s2, size_t n);

     wchar_t *
     wmemmove(wchar_t *s1, const wchar_t *s2, size_t n);

     wchar_t *
     wmemset(wchar_t *s, wchar_t c, size_t n);

     wchar_t *
     wcpcpy(wchar_t *s1, wchar_t *s2);

     wchar_t *
     wcpncpy(wchar_t *s1, wchar_t *s2, size_t n);

     int
     wcscasecmp(const wchar_t *s1, const wchar_t *s2);

     wchar_t *
     wcscat(wchar_t * restrict s1, const wchar_t * restrict s2);

     wchar_t *
     wcschr(const wchar_t *s, wchar_t c);

     int
     wcscmp(const wchar_t *s1, const wchar_t *s2);

     wchar_t *
     wcscpy(wchar_t * restrict s1, const wchar_t * restrict s2);

     size_t
     wcscspn(const wchar_t *s1, const wchar_t *s2);

     wchar_t *
     wcsdup(const wchar_t *s);

     size_t
     wcslcat(wchar_t *s1, const wchar_t *s2, size_t n);

     size_t
     wcslcpy(wchar_t *s1, const wchar_t *s2, size_t n);

     size_t
     wcslen(const wchar_t *s);

     int
     wcsncasecmp(const wchar_t *s1, const wchar_t *s2, size_t n);

     wchar_t *
     wcsncat(wchar_t * restrict s1, const wchar_t * restrict s2, size_t n);

     int
     wcsncmp(const wchar_t *s1, const wchar_t * s2, size_t n);

     wchar_t *
     wcsncpy(wchar_t * restrict s1, const wchar_t * restrict s2, size_t n);

     size_t
     wcsnlen(const wchar_t *s, size_t maxlen);

     wchar_t *
     wcspbrk(const wchar_t *s1, const wchar_t *s2);

     wchar_t *
     wcsrchr(const wchar_t *s, wchar_t c);

     size_t
     wcsspn(const wchar_t *s1, const wchar_t *s2);

     wchar_t *
     wcsstr(const wchar_t * restrict s1, const wchar_t * restrict s2);

DESCRIPTION

     The functions implement string manipulation operations over wide character strings.  For a detailed description, refer to documents for the
     respective single-byte counterpart, such as memchr(3).

SEE ALSO

     memchr(3), memcmp(3), memcpy(3), memmove(3), memset(3), stpcpy(3), stpncpy(3), strcasecmp(3), strcat(3), strchr(3), strcmp(3), strcpy(3),
     strcspn(3), strdup(3), strlcat(3), strlcpy(3), strlen(3), strncat(3), strncmp(3), strncpy(3), strnlen(3), strpbrk(3), strrchr(3), strspn(3),
     strstr(3)

STANDARDS

     These functions conform to ISO/IEC 9899:1999 (``ISO C99''), with the exception of wcpcpy(), wcpncpy(), wcscasecmp(), wcsdup(), wcsncasecmp(),
     and wcsnlen(), which conform to IEEE Std 1003.1-2008 (``POSIX.1''); and wcslcat() and wcslcpy(), which are extensions.

BSD
								   March 4, 2009							       BSD