Programming

My OS (Debian) and gcc use the UTF-8 locale. This code says that the char size is 1 byte but the size of 'a' is really 4 bytes.

Code:

int main(void)
{
    setlocale(LC_ALL, "en_US.UTF-8");

    printf("Char size: %i\nSize of char 'a': %i\nSize of Euro sign '€': %i\nLength of Euro sign: %i\n", sizeof(char), sizeof('a'), sizeof('€'), strlen("€"));

    return 0;
}

How can I make it output this which is more logical. I modified CHAR_BIT in limits.h to 32 but it still outputs 1 byte. Did I do something wrong ?

AAGH. Don't modify limits.h! You will break the compiler.

The size of a UTF-8 (8 bits) char is one byte.

You code has issues - sizeof(something) tells you how much space the compiler allocates for something, not the length of a character of a string. The answer is 1 for a single char.

mblen() gets the length of wide characters in bytes, use on wide characters.

Ah, that's why. I thought it would only apply to the program being compiled.
By default, yes, because in ASCII, 1 char = 1 byte, but making it equal to 32 bits (4 bytes) would be better suited for UTF-8 where chars can equal 4 bytes.



One of the things I wanted to point out: if 'a' is 4 bytes, isn't it overflowing char which is 1 byte ?

No. Because the type char is defined as being 1 byte. However, if the compiler thinks a character might be a multi-byte symbol, it might silently convert it to an character array, or a matching multi-byte type. If it's converted to an character array, sizeof will probably return the size of the pointer to it (int == 4 bytes), or the size of the internal type (Unicode = 32bit == 4 bytes). Which case matches I cannot tell, as I'm only doing educated guesses here.

The default value for a single char in C is usually promoted to an int. sizeof(int) = 4
but that does not mean that a single 'a' character actually is that large - 4 bytes.

A char in UTF-8 is 1 byte.

Your compiler stores a single char variable in the space it uses for an int. That is the bottom line for all the confusion.

BTW: CHAR_BIT by standard is currently defined to be fixed at 8, regardless of how the compiler decides to promote it. And regardless of what the locale may be. Which is why you should not change it to 32.

See C99 standards 5.2.4.2.1 Sizes of integer types <limits.h>

UTF8 is encoded in 8-bit segments but occasionally it can take more than one byte to encode a full character. So the number of characters is wholly unrelated to the number of characters in the string; changing a #define won't make it any more right. None of the usual strlen/sizeof/etc things apply to a system where character encodings aren't always the same size!

If you want to count the number of UTF-8 encoded characters in the string, as opposed to the physical length of the string in bytes, I once wrote code for a UTF8 getc. Use strlen to get the length of the datablock to feed it, then process UTF8 chars one by one until the end of the string to get the UTF8 char count.