I have a shell script - working perfectly on Oracle Linux - that detects the encoding (the charset to be exact) of the files in a specified directory using the "file" command (The file command outputs the charset in Linux, but doesn't do that in AIX), then if the file isn't a UTF-8 text file, it converts it to UTF-8 using "iconv" command.
I searched lots of forums and threads but it seems this is extremely hard to do in AIX, since the "file" command doesn't output the charset.
I also read this useful thread on this forum: Converting Unicode file to UTF8 format.
My problem is that if I want to use the "iconv" command to convert my files to UTF-8, how can I determine the charset of the original file ?
Code:
iconv -f FromCode -t ToCode
(The ToCode can be replaced by UTF-8, but I need to guess the FromCode).
Is there any way to do that ?
Does anyone have a working script on AIX that does what I want to do ?
Thank you,
Regards.
Moderator's Comments:
Please use CODE tags as required by forum rules!
Last edited by RudiC; 02-26-2018 at 11:29 AM..
Reason: Added CODE tags.
While working with russian text under FreeBSD&MySQL I need to convert a string from MySQL to the Unicode format.
I've just started my way in C++ under FreeBSD , so please explain me how can I get ascii code of Char variable and also how can i get a character into variable with the specified ascii... (3 Replies)
Hi, there:
I am relatively new to Unix. So, I am not even sure if I am asking is an easy or difficult task.
I want to peform GREP like command which will generate a list of files with a file format of UTF-8. I would especially like to know whether the files use UTF-8 or UTF-8N (in other... (0 Replies)
Hi,
I have a non-ascii character (Ŵ), which can be represented in UTF-8 encoding as equivalent hex value (\xC5B4). Is there a function in unix to convert this hex value back to display the charcter ? (10 Replies)
Hello,
I am trying to convert a 7bit ASCII file to UTF-8.
I have used iconv before though it can't recognize it for some reason and says unknown file encoding.
When I used ascii2uni package with different package, ./ascii2uni -a K -a I -a J -a X test_file > new_test_file
It still... (2 Replies)
Hello everyone!
I have a problem with printing ru_RU.UTF-8 from AIX using lp command.
#locale -a
C
POSIX
RU_RU.UTF-8
RU_RU
en_US.8859-15
en_US.ISO8859-1
en_US
ru_RU.ISO8859-5
ru_RU
#locale
LANG=en_US.UTF-8
LC_COLLATE=RU_RU.UTF-8
LC_CTYPE=RU_RU.UTF-8
LC_MONETARY="en_US" (3 Replies)
I have two Macs running 10.7.5.
We download .txt files from remote site to these local Macs using 'rsync -e ssh -avz...'.
The files on Mac1 are in the required format of pure UTF-8. The files on Mac2 are in UTF-8 (no BOM) which is wrong format for us; these formats are indicated using BBEdit.... (1 Reply)
Hi,
I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine
unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt
and i am getting some chinese characters as below which l opened the converted file on windows machine.
LANG=en_US.UTF-8... (3 Replies)
Hello Gang
Can you please help me in installing EN_GB UTF-8 on AIX 5.3 .
I have worked on Solaris and Linux, but not familier with AIX so hoping if someone show the roadmap. (5 Replies)
Hello all
i have utf-8 file that i try to convert to WINDOWS-1251 on linux
without any success
the file name is utf-8 when i try to do :
file -bi test.txt
it gives me :
text/plain; charset=utf-8
when i try to convert the file i do :
/usr/bin/iconv -f UTF-8 -t WINDOWS-1251 test.txt >... (1 Reply)
I am trying to develop a script which will work on a source UTF-8 file and perform one or more of the following
It will accept the target encoding as an argument e.g. US-ASCII or ISO-8859-1, etc
1. It should replace all occurrences of characters outside target character set by " " (space) or... (3 Replies)
Discussion started by: hemkiran.s
3 Replies
LEARN ABOUT SUNOS
iconv_zh
iconv_zh(5) File Formats Manual iconv_zh(5)NAME
iconv_zh - code set conversion tables
DESCRIPTION
The following code set conversions are supported:
Code Set Conversions Supported
Code Symbol TargetCode Symbol
GB 2312-80 zh_CN.euc UTF-8 UTF-8
GB 2312-80 zh_CN.euc HZ-GB-2312 HZ-GB-2312
GB 2312-80 zh_CN.euc ISO-2022-CN zh_CN.iso2022-CN
GBK zh_CN.gbk UTF-8 UTF-8
GBK zh_CN.gbk HZ-GB-2312 HZ-GB-2312
GBK zh_CN.gbk ISO-2022-CN zh_CN.iso2022-CN
GB18030-2000 zh_CN.gb18030 UTF-8 UTF-8
GB18030-2000 zh_CN.gb18030 IS0-2022-CN zh_CN.iso2022-CN
HZ-GB-2312 HZ-GB-2312 UTF-8 UTF-8
HZ-GB-2312 HZ-GB-2312 GB 2312-80 zh_CN.euc
HZ-GB-2312 HZ-GB-2312 GBK zh_CN.gbk
ISO-2022-CN zh_CN.iso2022-CN GB 2312-80 zh_CN.euc
ISO-2022-CN zh_CN.iso2022-CN GBK zh_CN.gbk
ISO-2022-CN zh_CN.iso2022-CN GB18030-2000 zh_CN.gb18030
IS0-2022-CN zh_CN.iso2022-CN UTF-8 UTF-8
Code Page 935 zh_CN-cp935 UTF-8 UTF-8
UTF-8 UTF-8 GB 2312-80 zh_CN.euc
UTF-8 UTF-8 GBK zh_CN.gbk
UTF-8 UTF-8 GB18030-2000 zh_CN.gb18030
UTF-8 UTF-8 HZ-GB-2312 HZ-GB-2312
UTF-8 UTF-8 IS0-2022-CN zh_CN.iso2022-CN
UTF-8 UTF-8 Code Page 935 zh_CN-cp935
GB 2312-80 zh_CN.euc Big-5 zh_TW-big5
GBK zh_CN.gbk Big-5 zh_TW-big5
GBK zh_CN.gbk Big-5p zh_TW-big5p
GBK zh_CN.gbk BIG5HK zh_HK-big5hk
GB18030-2000 zh_CN.gb18030 Big-5 zh_TW-big5
GB18030-2000 zh_CN.gb18030 Big-5p zh_TW-big5p
GB18030-2000 zh_CN.gb18030 BIG5HK zh_HK-big5hk
Big-5 zh_TW-big5 GB18030-2000 zh_CN.gb18030
Big-5 zh_TW-big5 GBK zh_CN.gbk
Big-5 zh_TW-big5 GB 2312-80 zh_CN.euc
Big-5p zh_TW-big5p GB18030-2000 zh_CN.gb18030
Big-5p zh_TW-big5p GBK zh_CN.gbk
BIG5HK zh_HK-big5hk GB18030-2000 zh_CN.gb18030
BIG5HK zh_HK-big5hk GBK zh_CN.gbk
Conversion are performed as described below, For all conver- sions, if the source code set includes characters not inclu- ded in the target
code set, conversion and output for all such characters will be done using a substitute characters.
zh_CN.euc to UTF-8 and UTF-8 to zh_CN.euc
Conversion modules can be used to convert GB 2312-80 characters between EUC-CN and UTF-8 encodings. If input data which does not have the
corresponding character in the target code is encountered, it will be replaced with the substitute character (zh_CN.euc: '??' (0x3f3f),
UTF-8: U+FFFD(0xefbfbd)).
zh_CN.gbk to UTF-8 and UTF-8 to zh_CN.gbk
Conversion modules can be used to convert GBK characters between GBK and UTF-8 encodings. If input data which does not have the correspond-
ing character in the target code is encountered, it will be replaced with the subsitute character (zh_CN.gbk: '??' (0x3f3f), UTF-8:
U+FFFD(0xefbfbd)).
zh_CN.gb18030 to UTF-8 and UTF-8 to zh_CN.gb18030
Conversion modules can be used to convert GB18030-2000 characters between GB18030 and UTF-8 encodings. If input data which does not have
the corresponding character in the target code is encountered, it will be replaced with the substitute character (zh_CN.GB18030: '??'
(0x3f3f), UTF-8: U+FFFD(0xefbfbd)).
zh_CN.euc to zh_CN.iso2022-CN and zh_CN.iso2022-CN to zh_CN.euc
Conversion modules are provided to convert GB 2312-80 characters between IS0-2022-CN and EUC-CN encodings.
zh_CN.gbk to zh_CN.iso2022-CN and zh_CN.iso2022-CN to zh_CN.euc
Conversion modules are provided to convert GB 2312-80, CNS 11643-1992 plane 1 and 2 characters between ISO-2022-CN and EUC-CN encoding. If
input data which doesn't belong to above charset is encountered, it will be replaced with the substitute character (zh_CN.gbk: 0xa1f5,
zh_CN.iso2022-CN: 0x2175).
zh_CN.euc to HZ-GB-2312 and HZ-GB-2312 to zh_CN.euc
Conversion modules are provided to convert GB 2312-80 characters between EUC-CN and HZ encodings.
zh_CN.gbk to HZ-GB-2312 and HZ-GB-2312 to zh_CN.gbk
Conversion modules are provided to convert GB 2312-80 characters between EUC-CN and HZ encodings. If input data which does not belong to
the above charset is encountered, it will be replaced with the substitute characters (HZ-GB-2312: 0x2175).
zh_CN.iso2022-CN to UTF-8 and UTF-8 to zh_CN.iso2022-CN
Conversion modules are provided to convert GB 2312-80 and CNS 11643 plane 1 and 2 characters between ISO-2022-CN and UTF-8 encodings. If
input data which does not belong to the above charsets is encountered, it will be replaced with the substitute characters
(zh_CN.iso2022-CN: '??' (0x3f3f), UTF-8: U+FFFD).
HZ-GB-2312 to UTF-8 and UTF-8 to HZ-GB-2312
Conversion modules can be used to convert GB 2312-80 characters between HZ and UTF-8 encoding. If input data which does not belong to the
above charset is encountered, it will be replaced with the substitute characters ( HZ-GB-2312: '??' (0x3f3f), UTF-8: U+FFFD).
zh_CN-cp935 to UTF-8 and UTF-8 to zh_CN-cp935
Conversion modules are provided to convert GB 2312-80 characters between IBM Code Page 935 and UTF-8 encodings.
zh_CN.euc to zh_TW-big5 and zh_TW-big5 to zh_CN.euc
Conversion modules are provided to convert between simplified chinese (EUC-CN encoding) and traditional chinese (BIG5 encoding). If input
data which does not have corresponding simplified/traditional chinese character is encountered, it will be replaced with the substitute
character (zh_CN.euc: '__' (0x5f5f), zh_TW-big5: '__' (0x5f5f)).
zh_CN.gbk to zh_TW-big5 and zh_TW-big5 to zh_CN.gbk
Conversion modules are provided to convert GBK characters between GBK and BIG5 encoding. If input data which neither has corresponding sim-
plified/traditional chinese character nor belongs to the other charset is encountered, it will be replaced with the substitute character
(zh_CN.gbk: '__' (0x5f5f), zh_TW-big5: '__' (0x5f5f)).
zh_CN.gbk to zh_TW-big5p and zh_TW-big5p to zh_CN.gbk
Conversion modules can be used to convert GBK or Big-5p characters between GBK and Big-5p encodings. If input data which does not belong to
the other charset, it will be replaced with the substitute character (zh_CN.gbk: '__' (0x5f5f), zh_TW-big5p: '__' (0x5f5f)).
zh_CN.gbk to zh_HK-big5hk and zh_HK-big5hk to zh_CN.gbk
Conversion modules can be used to convert GBK characters between GBK and BIG5HK encodings. if input data which does not belong to the other
charset, it will be replaced with the substitute character (zh_CN.gbk: '__' (0x5f5f), zh_HK-big5hk: '__' (0x5f5f)).
SEE ALSO iconv(1), iconv(3), iconv(5), iconv_zh_TW(5)
2 Nov 2001 iconv_zh(5)
02-26-2018
