UNIX for Beginners Questions & Answers

Dears,

I have a shell script - working perfectly on Oracle Linux - that detects the encoding (the charset to be exact) of the files in a specified directory using the "file" command (The file command outputs the charset in Linux, but doesn't do that in AIX), then if the file isn't a UTF-8 text file, it converts it to UTF-8 using "iconv" command.

I searched lots of forums and threads but it seems this is extremely hard to do in AIX, since the "file" command doesn't output the charset.

I also read this useful thread on this forum: Converting Unicode file to UTF8 format.

My problem is that if I want to use the "iconv" command to convert my files to UTF-8, how can I determine the charset of the original file ?

Code:

iconv -f FromCode -t ToCode

(The ToCode can be replaced by UTF-8, but I need to guess the FromCode).

Is there any way to do that ?

Does anyone have a working script on AIX that does what I want to do ?

Thank you,
Regards.

Moderator's Comments:

Please use CODE tags as required by forum rules!

If you don't know what codeset was used to encode a file, there isn't much that can be done to guess at what it might be.
It is easy to guess that it is just ASCII if there aren't any bytes with the high order bit set and there aren't any NUL bytes. It is easy to guess that it might be UTF-16 if every other byte is a NUL byte. Guessing that some text might be encoded in one of the EBCDIC codesets might not be too hard, but correctly guessing which variant is another matter. And, other than that, good luck. The differences between the various 8859-* character sets is only obvious to most people if you know what the text in the file is supposed to be beforehand.

Hi,

the file command uses a file called magic to identify the file type. According to the POSIX man page the -m flag can be used to specify an own magic file and I think the AIX file command supports this flag too.
Maybe you can obtain or create a magic file that fits your needs.

Hi cero,
The file magic file is used to identify things like executable file formats (that have certain fixed binary values at fixed locations in a file). It is great for various a.out files, music files, photographic files, PDF files, word processing files, spreadsheet files, and similar things with fixed headers.

The file utility uses other built-in knowledge when trying to identify the language or codeset used in a text file. It sounds like the Linux file utility has some code built-in that does a better job of guessing at codesets underlying a text file than the AIX file utility for the files that JeanM-1 is processing. Whether or not the GNU file utility source would build correctly on AIX is something JeanM-1 may want to investigate. But, copying a Linux magic file to AIX and having the AIX file utility use it instead of AIX's default magic file isn't likely to make any difference for this issue.

Hi Don,
thanks for pointing out that my post was not that clear - I did not suggest to copy the Linux magic file to AIX (and am nearly sure that it would not work), but my post can be interpreted like that.
The magic file would be my starting point if I'd have to solve JeanM-1's problem (and the Byte Order Mark the first thing I'd start playing around with to see if I get anywhere).