Charsets and encoding details


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Charsets and encoding details
# 1  
Old 11-14-2012
Charsets and encoding details

Hello gurus, I would like to get deep into charset and encoding isse, also tried google it but no luck. Please see bellow

My configuration
Code:
[pista@HP-PC MULTIBOOT]$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

I have file1, containing text. This text I am able to see correctly only on M$ windows, If i just open the file with less, cat or vi I get this:
Code:
[pista@HP-PC konvertovanie]$ cat file1 
- Prich�dzaj�.
- Kto prich�dza?
N�� svet okupuj
vyvinut� �udsk� druhy,

[pista@HP-PC konvertovanie]$ less file1 
- Prich<E1>dzaj<FA>.
- Kto prich<E1>dza?
N<E1><9A> svet okupuj<FA>
vyvinut<E9> <BE>udsk<E9> druhy,

[pista@HP-PC konvertovanie]$ vi file1 
- Prichádzajú.
- Kto prichádza?
Ná<9a> svet okupujú
vyvinuté ľudské druhy,

Under linux I have to use iconv to see it correctly
Code:
[pista@HP-PC konvertovanie]$ iconv -f WINDOWS-1250 -t UTF-8 file1
- Prichádzajú.
- Kto prichádza?
Náš svet okupujú
vyvinuté ľudské druhy,

I understand that this is because of that file was coded in one format (WINDOWS-1250) and encoded in another (UTF-8). But can you clarify the following?

1.) When I check the decimal ASCII value of each character I get following lines. So what does negative values mean and what is that code 341 (instead of á) ? AFAIK ASCII is from 0-127.
Code:
[pista@HP-PC konvertovanie]$ cat file1 | od -An -t dC -c
   45   32   80  114  105   99  104  -31  100  122   97  106   -6   46   13   10
    -         P    r    i    c    h  341    d    z    a    j  372    .   \r   \n
   45   32   75  116  111   32  112  114  105   99  104  -31  100  122   97   63
    -         K    t    o         p    r    i    c    h  341    d    z    a    ?
   13   10   78  -31 -102   32  115  118  101  116   32  111  107  117  112  117
   \r   \n    N  341  232         s    v    e    t         o    k    u    p    u
  106   -6   13   10   48   48   58   48   48   58   48   53   44   56   50   48
    j  372   \r   \n    0    0    :    0    0    :    0    5    ,    8    2    0
   32   45   45   62   32   48   48   58   48   48   58   48   55   44   54   53
         -    -    >         0    0    :    0    0    :    0    7    ,    6    5
   52   13   10  118  121  118  105  110  117  116  -23   32  -66  117  100  115
    4   \r   \n    v    y    v    i    n    u    t  351       276    u    d    s
  107  -23   32  100  114  117  104  121   44   13   10
    k  351         d    r    u    h    y    ,   \r   \n

2.) My assumption is that if UTF-8 and WINDOWS-1250 uses for same characters different "numbers" (code representation) then if some character will be encoded using encoding1 (WINDOWS-1250) it gains approporiate "code1" from encoding1 table. So if this encoded character (or more likely it's number representation, which is "code1") will be decoded using another encoding (UTF-8) the only thing that happens here is that for "code1" there will be lookup in encoding2 (UTF-8) table and approporiate character from encoding2 table is asigned, am I right ? I think after some exaple it will be clear:

Please look at following sites, they shows what will happend if you encode with one encoding and decode with another. Seems that until you reach 127 (decimal) boundary no mather if you decode with wrong decoding (this is why some characters in above example was displayed correctly even when wrong encoding was used).

from UTF-8 to WINDOWS-1250
Encoding utf-8 to windows-1250

from WINDOWS-1250 to UTF-8
Encoding windows-1250 to utf-8

According this site The extreme UTF-8 table the "á" character is encoded in UTF-8 as a 225. According wikipedia Windows-1250 - Wikipedia, the free encyclopedia "á" has also value 225 in Windows-1250. So why is "á" not dispplayed correctly even if I use wrong encoding, check here and type "á" Encoding / decoding tool. Analyze character encoding problems and errors. ? Also some interesting observation, in UTF-8 table there is "š" character two times (one time with 154 and another with 453 code) why ?

3.) If i understand it right there is no way to tell how file was encoded (unless there is some header that specify this, or you do some statistical language analysis etc.). So why/how "file" commands recognize UTF-8 encoding but not WINDOWS-1250 ?
Code:
[pista@HP-PC konvertovanie]$ file -bi file1 
text/plain; charset=unknown-8bit
[pista@HP-PC konvertovanie]$ iconv -f WINDOWS-1250 -t UTF-8 file1 > file1.utf8
[pista@HP-PC konvertovanie]$ file -bi file1.utf8 
text/plain; charset=utf-8

Thank you very much
# 2  
Old 11-15-2012
Quote:
Originally Posted by wakatana
Hello gurus, I would like to get deep into charset and encoding isse, also tried google it but no luck. Please see bellow

My configuration
Code:
[pista@HP-PC MULTIBOOT]$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

I have file1, containing text. This text I am able to see correctly only on M$ windows, If i just open the file with less, cat or vi I get this:
Code:
[pista@HP-PC konvertovanie]$ cat file1 
- Prich�dzaj�.
- Kto prich�dza?
N�� svet okupuj
vyvinut� �udsk� druhy,

[pista@HP-PC konvertovanie]$ less file1 
- Prich<E1>dzaj<FA>.
- Kto prich<E1>dza?
N<E1><9A> svet okupuj<FA>
vyvinut<E9> <BE>udsk<E9> druhy,

[pista@HP-PC konvertovanie]$ vi file1 
- Prichádzajú.
- Kto prichádza?
Ná<9a> svet okupujú
vyvinuté ľudské druhy,

Under linux I have to use iconv to see it correctly
Code:
[pista@HP-PC konvertovanie]$ iconv -f WINDOWS-1250 -t UTF-8 file1
- Prichádzajú.
- Kto prichádza?
Náš svet okupujú
vyvinuté ľudské druhy,

I understand that this is because of that file was coded in one format (WINDOWS-1250) and encoded in another (UTF-8). But can you clarify the following?

1.) When I check the decimal ASCII value of each character I get following lines. So what does negative values mean and what is that code 341 (instead of á) ? AFAIK ASCII is from 0-127.
Code:
[pista@HP-PC konvertovanie]$ cat file1 | od -An -t dC -c
   45   32   80  114  105   99  104  -31  100  122   97  106   -6   46   13   10
    -         P    r    i    c    h  341    d    z    a    j  372    .   \r   \n
   45   32   75  116  111   32  112  114  105   99  104  -31  100  122   97   63
    -         K    t    o         p    r    i    c    h  341    d    z    a    ?
   13   10   78  -31 -102   32  115  118  101  116   32  111  107  117  112  117
   \r   \n    N  341  232         s    v    e    t         o    k    u    p    u
  106   -6   13   10   48   48   58   48   48   58   48   53   44   56   50   48
    j  372   \r   \n    0    0    :    0    0    :    0    5    ,    8    2    0
   32   45   45   62   32   48   48   58   48   48   58   48   55   44   54   53
         -    -    >         0    0    :    0    0    :    0    7    ,    6    5
   52   13   10  118  121  118  105  110  117  116  -23   32  -66  117  100  115
    4   \r   \n    v    y    v    i    n    u    t  351       276    u    d    s
  107  -23   32  100  114  117  104  121   44   13   10
    k  351         d    r    u    h    y    ,   \r   \n

2.) My assumption is that if UTF-8 and WINDOWS-1250 uses for same characters different "numbers" (code representation) then if some character will be encoded using encoding1 (WINDOWS-1250) it gains approporiate "code1" from encoding1 table. So if this encoded character (or more likely it's number representation, which is "code1") will be decoded using another encoding (UTF-8) the only thing that happens here is that for "code1" there will be lookup in encoding2 (UTF-8) table and approporiate character from encoding2 table is asigned, am I right ? I think after some exaple it will be clear:

Please look at following sites, they shows what will happend if you encode with one encoding and decode with another. Seems that until you reach 127 (decimal) boundary no mather if you decode with wrong decoding (this is why some characters in above example was displayed correctly even when wrong encoding was used).

from UTF-8 to WINDOWS-1250
Encoding utf-8 to windows-1250

from WINDOWS-1250 to UTF-8
Encoding windows-1250 to utf-8

According this site The extreme UTF-8 table the "á" character is encoded in UTF-8 as a 225. According wikipedia Windows-1250 - Wikipedia, the free encyclopedia "á" has also value 225 in Windows-1250. So why is "á" not dispplayed correctly even if I use wrong encoding, check here and type "á" Encoding / decoding tool. Analyze character encoding problems and errors. ? Also some interesting observation, in UTF-8 table there is "š" character two times (one time with 154 and another with 453 code) why ?

3.) If i understand it right there is no way to tell how file was encoded (unless there is some header that specify this, or you do some statistical language analysis etc.). So why/how "file" commands recognize UTF-8 encoding but not WINDOWS-1250 ?
Code:
[pista@HP-PC konvertovanie]$ file -bi file1 
text/plain; charset=unknown-8bit
[pista@HP-PC konvertovanie]$ iconv -f WINDOWS-1250 -t UTF-8 file1 > file1.utf8
[pista@HP-PC konvertovanie]$ file -bi file1.utf8 
text/plain; charset=utf-8

Thank you very much
Most of the websites you referenced are not really applicable to this question. By convention, using LANG=en_US.UTF-8 indicates that you are using the English language with formatting conventions used in the United States encoded using the UTF-8 character set.

The UTF-8 character set encodes over a million characters using one to six bytes to encode a character. Any byte that has the high order bit set is part of a multi-byte character. Any byte that has the high order bit clear represents the same character as the US-ASCII character with the same value. See Wikipedia UTF-8 for details.

The Windows-1250 codeset is used to represent text for Central and Eastern European characters using Latin script and encodes 251 characters using one byte to encode a character. Any byte that has the high order bit clear represents the same character as the US-ASCII character with the same value. See Wikipedia Windows-1250 for details. That web page shows that the character with decimal value 251 in Windows-1250 ("á") corresponds to the Unicode character with value 00E1 which is encoded in UTF-8 as the two byte sequence 0xC3 0xA1 (or in decimal 195 161). Since you can see that the single byte with value 251 (base 10) in Windows-1250 is not the same as the two byte values 195 followed by 161 in UTF-8, you will not see the same printed characters when you try to look at Windows-1250 codeset characters when you tell the system that you are using a locale with a UTF-8 codeset representation of characters. Furthermore, in UTF-8 there is never a single byte character with the high bit set and there is never a multi-byte character than has any byte without the high bit set.

The iconv utility knows how to convert characters from one codeset to another codeset and as you have seen does so successfully. But, expecting characters from a single byte codeset with the high bit set to be magically converted from invalid characters in UTF-8 based on an unspecified assumption that invalid UTF-8 characters should be treated as Windows-1250 characters just won't work.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Solaris

View file encoding then change encoding.

Hi all!! I´m using command file -i myfile.xml to validate XML file encoding, but it is just saying regular file . I´m expecting / looking an output as UTF8 or ANSI / ASCII Is there command to display the files encoding? Thank you! (2 Replies)
Discussion started by: mrreds
2 Replies

2. UNIX for Dummies Questions & Answers

at -l doesnt give details of the scheduled job. How to get the details?

I have scheduled couple of shell scripts to run using 'at' command. The o/p of at -l is: $ at -l 1320904800.a Thu Nov 10 01:00:00 2011 1320894000.a Wed Nov 9 22:00:00 2011 1320876000.a Wed Nov 9 17:00:00 2011 $ uname -a SunOS dc2prcrptetl2 5.9 Generic_122300-54 sun4u sparc... (2 Replies)
Discussion started by: superparticle
2 Replies

3. Solaris

Encoding problem

Hi All, Hope you can help me with the below :). I'm working on a script on SUN solaris and I'm facing a problem with the number encoding as shown below, 1 is encoded to 31 (this is ASCII so it's ok) 11 is encoded as B118 !!! don't know why 111 is encoded as B1580C !!! don't know why ... (4 Replies)
Discussion started by: /dev/bag
4 Replies

4. Shell Programming and Scripting

How to find the file encoding and updating the file encoding?

Hi, I am beginner to Unix. My requirement is to validate the encoding used in the incoming file(csv,txt).If it is encoded with UTF-8 format,then the file should remain as such otherwise i need to chnage the encoding to UTF-8. Please advice me how to proceed on this. (7 Replies)
Discussion started by: cnraja
7 Replies

5. UNIX for Dummies Questions & Answers

Encoding Type

Hi, Where can I find the encoding type in a unix server ? Thanks in advance !!! (1 Reply)
Discussion started by: risshanth
1 Replies

6. Shell Programming and Scripting

Encoding troubles

Hello All I have a set of files, each one containing some lines that follows that regex: regex='disabled\,.*\,\".*\"'and here is what file says about each files: file <random file> <random file> ASCII text, with CRLF line terminatorsSo, as an example, here is what a file ("Daffy Duck - The... (3 Replies)
Discussion started by: tukuyomi
3 Replies

7. Shell Programming and Scripting

Araic Encoding

hi folks , I have a shell script which contain SQL query that dump some data from the DB in arabic and this data is written to a file in unix machine but the problem that the arabic data is appear like ??????????|111|???????? even when I move it to my windows XP machine. Any one have an Idea... (2 Replies)
Discussion started by: habuzahra
2 Replies

8. Shell Programming and Scripting

get the file encoding

Hello! The system is AIX 5.3 Give please command or script to get the file encoding Thanks (2 Replies)
Discussion started by: vinment
2 Replies

9. Shell Programming and Scripting

URL encoding

Hi All, I want to do URL encoding using shell script in my project. I decided that the sed is the correct tool to do this. But I am unable achieve what I wanted using sed. kindly help me to get rid of this. My requirement is , there will be one URL with all special character, spaces etc... ... (8 Replies)
Discussion started by: Vichu
8 Replies

10. UNIX for Dummies Questions & Answers

encoding

Hi, I'm using putty and when I try to write ü it writes | (or when I try to write é , it writes i) I tried to change settings/translation of putty but with no success I have KSH # locale LANG= LC_CTYPE="C" LC_NUMERIC="C" LC_TIME="C" LC_COLLATE="C" LC_MONETARY="C" LC_MESSAGES="C"... (3 Replies)
Discussion started by: palmer18
3 Replies
Login or Register to Ask a Question