Linux

Hi,

I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine

Code:

unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt

and i am getting some chinese characters as below which l opened the converted file on windows machine.

Code:

LANG=en_US.UTF-8
਍䰀䌀开䌀吀夀倀䔀㴀∀攀渀开唀匀⸀唀吀䘀ⴀ㠀∀ഀഀ
LC_NUMERIC="en_US.UTF-8"
਍䰀䌀开吀䤀䴀䔀㴀∀攀渀开唀匀⸀唀吀䘀ⴀ㠀∀ഀഀ
LC_COLLATE="en_US.UTF-8"
਍䰀䌀开䴀伀一䔀吀䄀刀夀㴀∀攀渀开唀匀⸀唀吀䘀ⴀ㠀∀ഀഀ
LC_MESSAGES="en_US.UTF-8"
਍䰀䌀开倀䄀倀䔀刀㴀∀攀渀开唀匀⸀唀吀䘀ⴀ㠀∀ഀഀ
LC_NAME="en_US.UTF-8"
਍䰀䌀开䄀䐀䐀刀䔀匀匀㴀∀攀渀开唀匀⸀唀吀䘀ⴀ㠀∀ഀഀ
LC_TELEPHONE="en_US.UTF-8"
਍䰀䌀开䴀䔀䄀匀唀刀䔀䴀䔀一吀㴀∀攀渀开唀匀⸀唀吀䘀ⴀ㠀∀ഀഀ
LC_IDENTIFICATION="en_US.UTF-8"
਍䰀䌀开䄀䰀䰀㴀ഀഀ

This is just a test file i was working on but the actual file contains numbers of rows and columns. Am i missing anything above up here?

The requirements are for the output should be as below

- UTF-16 Little endian
- preceded with a byte order marker --ff and fe
- Windows line endings

Any pointers will be great

Thanks,
P

Usually all the file transferred should be binary format so that nothing can be changed so that we don’t get any unknown characters

Let us consider the file name as Orgdata_UTF8.txt then output file as Orgdata.txt

Code:

unix2dos < Orgdata_UTF8.txt |iconv -f UTF-8 -t UTF-16LE>Orgdata.txt

As some systems add the BOM by default and some systems doesn’t add the BOM based on the operating systems and it is also the similar case with UTF-16LE format as it is sometimes recognised as UTF-16 and some as UTF-16LE based on versions and use them as needed.

Adding BOM manually

Create a new file as below Orgdata.txt and check the file type using file command to confirm that it is UTF-16LE format and then convert it as below

Code:

printf "\xff\xfe" > Orgdata.txt
file Orgdata.txt 
unix2dos < Orgdata_UTF8.txt |iconv -f UTF-8 -t UTF-16LE>>Orgdata.txt

Use the hex coder to check if you have got the desired result of ff fe or not. This result varies depending on the type of hexdump used.

Code:

cat Orgdata.txt |hexdump |less  -- this shows as fe ff          
xxd < Orgdata.txt |less              -- same file shows as ff fe

Code:

cat -vT Orgdata.txt

In reality both of them are same as one of them shows the reversing output.

This has resolved my issue

iconv and BOMs are a gray area in the Unicode specification. A useful discussion regarding iconv and presence or lack of a BOM is here

Thanks for the reply fpmurphy. I have already gone thru that link in getting the desired result.

Anyway thanks for the help.

Thanks,
P