Shell Programming and Scripting

Hi Don,

Below is the output

Code:

od -bc filename |head -3
0000000  456 567 043 000 123 000 456 000 145 000 157 000 184 000 184 000
           ▒   ▒   "  \0   R  \0   P  \0   e  \0   n  \0  Qt  \0   l  \0

I have tried both the command suggested by you.

1. iconv -f UTF-16 -t UTF-8 filename > utf8.txt .. got converted in a starnge format

Code:

head -1 utf8.txt
琀 挀漀搀攀 漀昀 ⠀　⤀⸀ഀ਀∀一椀最栀琀开䨀

2. iconv -f UTF-16 -t ISO8859-1 filename > 8859.txt this is giving nothing rather than a empty file.

About sensitivity of data yes data is being altered fully pasting just the dummy samples.

Hi Riverstone,

It might be worth a look here, to see if you can identify the file type.

Regards

Dave

I am seriously surprised by your od -bc output, as 456, 567, and 184 cannot be octal bytes, and 456 in the beginning is interpreted as a block char, whilst it's a "P" later on. 184 has two meanings as well. Puzzled.

And, it's different from previous samples again. How many files are we talking of?

And yet you still refuse to tell us what locale you're using! What is the output from the command?:

Code:

locale

File is surely UTF-16 only, we are talking about the single file.

I have heard that on some versions of AIX, the en_US locale uses the IBM 850 code set rather than USASCII, ISO 8859-1, or UTF-8 that would commonly be used on other systems.

Use:

Code:

locale -a|grep en_US

to get a list of available US English locales. Hopefully, you will see something like en_US.ISO8859-1 or en_US.UTF-8. If en_US.ISO8859-1 is in the list, try:

Code:

LC_ALL=en_US.ISO8859-1 cat 8859.txt

or if en_US.UTF-8 is in the list, try:

Code:

LC_ALL=en_US.US.UTF-8 cat utf8.txt

where 8859.txt and utf8..txt are the files you created earlier using iconv. You could also try setting LC_ALL=C before cat'ing those two files.

If one of those works, look through your shell's initialization scripts and change whatever is setting LANG and the LC_* variables to en_US to instead set them to the one of the above that works for you.