I have a shell script - working perfectly on Oracle Linux - that detects the encoding (the charset to be exact) of the files in a specified directory using the "file" command (The file command outputs the charset in Linux, but doesn't do that in AIX), then if the file isn't a UTF-8 text file, it converts it to UTF-8 using "iconv" command.
I searched lots of forums and threads but it seems this is extremely hard to do in AIX, since the "file" command doesn't output the charset.
I also read this useful thread on this forum: Converting Unicode file to UTF8 format.
My problem is that if I want to use the "iconv" command to convert my files to UTF-8, how can I determine the charset of the original file ?
(The ToCode can be replaced by UTF-8, but I need to guess the FromCode).
Is there any way to do that ?
Does anyone have a working script on AIX that does what I want to do ?
Thank you,
Regards.
Moderator's Comments:
Please use CODE tags as required by forum rules!
Last edited by RudiC; 02-26-2018 at 11:29 AM..
Reason: Added CODE tags.
If you don't know what codeset was used to encode a file, there isn't much that can be done to guess at what it might be.
It is easy to guess that it is just ASCII if there aren't any bytes with the high order bit set and there aren't any NUL bytes. It is easy to guess that it might be UTF-16 if every other byte is a NUL byte. Guessing that some text might be encoded in one of the EBCDIC codesets might not be too hard, but correctly guessing which variant is another matter. And, other than that, good luck. The differences between the various 8859-* character sets is only obvious to most people if you know what the text in the file is supposed to be beforehand.
the file command uses a file called magic to identify the file type. According to the POSIX man page the -m flag can be used to specify an own magic file and I think the AIX file command supports this flag too.
Maybe you can obtain or create a magic file that fits your needs.
Hi cero,
The file magic file is used to identify things like executable file formats (that have certain fixed binary values at fixed locations in a file). It is great for various a.out files, music files, photographic files, PDF files, word processing files, spreadsheet files, and similar things with fixed headers.
The file utility uses other built-in knowledge when trying to identify the language or codeset used in a text file. It sounds like the Linux file utility has some code built-in that does a better job of guessing at codesets underlying a text file than the AIX file utility for the files that JeanM-1 is processing. Whether or not the GNU file utility source would build correctly on AIX is something JeanM-1 may want to investigate. But, copying a Linux magic file to AIX and having the AIX file utility use it instead of AIX's default magic file isn't likely to make any difference for this issue.
Hi Don,
thanks for pointing out that my post was not that clear - I did not suggest to copy the Linux magic file to AIX (and am nearly sure that it would not work), but my post can be interpreted like that.
The magic file would be my starting point if I'd have to solve JeanM-1's problem (and the Byte Order Mark the first thing I'd start playing around with to see if I get anywhere).
I am trying to develop a script which will work on a source UTF-8 file and perform one or more of the following
It will accept the target encoding as an argument e.g. US-ASCII or ISO-8859-1, etc
1. It should replace all occurrences of characters outside target character set by " " (space) or... (3 Replies)
Hello all
i have utf-8 file that i try to convert to WINDOWS-1251 on linux
without any success
the file name is utf-8 when i try to do :
file -bi test.txt
it gives me :
text/plain; charset=utf-8
when i try to convert the file i do :
/usr/bin/iconv -f UTF-8 -t WINDOWS-1251 test.txt >... (1 Reply)
Hello Gang
Can you please help me in installing EN_GB UTF-8 on AIX 5.3 .
I have worked on Solaris and Linux, but not familier with AIX so hoping if someone show the roadmap. (5 Replies)
Hi,
I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine
unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt
and i am getting some chinese characters as below which l opened the converted file on windows machine.
LANG=en_US.UTF-8... (3 Replies)
I have two Macs running 10.7.5.
We download .txt files from remote site to these local Macs using 'rsync -e ssh -avz...'.
The files on Mac1 are in the required format of pure UTF-8. The files on Mac2 are in UTF-8 (no BOM) which is wrong format for us; these formats are indicated using BBEdit.... (1 Reply)
Hello everyone!
I have a problem with printing ru_RU.UTF-8 from AIX using lp command.
#locale -a
C
POSIX
RU_RU.UTF-8
RU_RU
en_US.8859-15
en_US.ISO8859-1
en_US
ru_RU.ISO8859-5
ru_RU
#locale
LANG=en_US.UTF-8
LC_COLLATE=RU_RU.UTF-8
LC_CTYPE=RU_RU.UTF-8
LC_MONETARY="en_US" (3 Replies)
Hello,
I am trying to convert a 7bit ASCII file to UTF-8.
I have used iconv before though it can't recognize it for some reason and says unknown file encoding.
When I used ascii2uni package with different package, ./ascii2uni -a K -a I -a J -a X test_file > new_test_file
It still... (2 Replies)
Hi,
I have a non-ascii character (Ŵ), which can be represented in UTF-8 encoding as equivalent hex value (\xC5B4). Is there a function in unix to convert this hex value back to display the charcter ? (10 Replies)
Hi, there:
I am relatively new to Unix. So, I am not even sure if I am asking is an easy or difficult task.
I want to peform GREP like command which will generate a list of files with a file format of UTF-8. I would especially like to know whether the files use UTF-8 or UTF-8N (in other... (0 Replies)
While working with russian text under FreeBSD&MySQL I need to convert a string from MySQL to the Unicode format.
I've just started my way in C++ under FreeBSD , so please explain me how can I get ascii code of Char variable and also how can i get a character into variable with the specified ascii... (3 Replies)