Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Convert files to UTF-8 on AIX 7.1 Post 303013712 by JeanM-1 on Monday 26th of February 2018 09:43:42 AM
Old 02-26-2018
Convert files to UTF-8 on AIX 7.1

Dears,

I have a shell script - working perfectly on Oracle Linux - that detects the encoding (the charset to be exact) of the files in a specified directory using the "file" command (The file command outputs the charset in Linux, but doesn't do that in AIX), then if the file isn't a UTF-8 text file, it converts it to UTF-8 using "iconv" command.

I searched lots of forums and threads but it seems this is extremely hard to do in AIX, since the "file" command doesn't output the charset.

I also read this useful thread on this forum: Converting Unicode file to UTF8 format.

My problem is that if I want to use the "iconv" command to convert my files to UTF-8, how can I determine the charset of the original file ?
Code:
iconv -f FromCode -t ToCode

(The ToCode can be replaced by UTF-8, but I need to guess the FromCode).

Is there any way to do that ?

Does anyone have a working script on AIX that does what I want to do ?

Thank you,
Regards.

Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!

Last edited by RudiC; 02-26-2018 at 11:29 AM.. Reason: Added CODE tags.
 

10 More Discussions You Might Find Interesting

1. Programming

Howto convert Ascii -> UTF-8 & back C++

While working with russian text under FreeBSD&MySQL I need to convert a string from MySQL to the Unicode format. I've just started my way in C++ under FreeBSD , so please explain me how can I get ascii code of Char variable and also how can i get a character into variable with the specified ascii... (3 Replies)
Discussion started by: macron
3 Replies

2. UNIX for Dummies Questions & Answers

Finding files with UTF-8 BOM

Hi, there: I am relatively new to Unix. So, I am not even sure if I am asking is an easy or difficult task. I want to peform GREP like command which will generate a list of files with a file format of UTF-8. I would especially like to know whether the files use UTF-8 or UTF-8N (in other... (0 Replies)
Discussion started by: kotoponus
0 Replies

3. UNIX for Advanced & Expert Users

Convert UTF-8 encoded hex value to a character

Hi, I have a non-ascii character (Ŵ), which can be represented in UTF-8 encoding as equivalent hex value (\xC5B4). Is there a function in unix to convert this hex value back to display the charcter ? (10 Replies)
Discussion started by: sumirmehta
10 Replies

4. Red Hat

Can't convert 7bit ASCII to UTF-8

Hello, I am trying to convert a 7bit ASCII file to UTF-8. I have used iconv before though it can't recognize it for some reason and says unknown file encoding. When I used ascii2uni package with different package, ./ascii2uni -a K -a I -a J -a X test_file > new_test_file It still... (2 Replies)
Discussion started by: rockf1bull
2 Replies

5. AIX

How to print UTF-8 from AIX (lp)

Hello everyone! I have a problem with printing ru_RU.UTF-8 from AIX using lp command. #locale -a C POSIX RU_RU.UTF-8 RU_RU en_US.8859-15 en_US.ISO8859-1 en_US ru_RU.ISO8859-5 ru_RU #locale LANG=en_US.UTF-8 LC_COLLATE=RU_RU.UTF-8 LC_CTYPE=RU_RU.UTF-8 LC_MONETARY="en_US" (3 Replies)
Discussion started by: burnAF
3 Replies

6. OS X (Apple)

Changing txt files to pure UTF-8

I have two Macs running 10.7.5. We download .txt files from remote site to these local Macs using 'rsync -e ssh -avz...'. The files on Mac1 are in the required format of pure UTF-8. The files on Mac2 are in UTF-8 (no BOM) which is wrong format for us; these formats are indicated using BBEdit.... (1 Reply)
Discussion started by: sovdia
1 Replies

7. Linux

Help to Convert file from UNIX UTF-8 to Windows UTF-16

Hi, I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt and i am getting some chinese characters as below which l opened the converted file on windows machine. LANG=en_US.UTF-8... (3 Replies)
Discussion started by: phanidhar6039
3 Replies

8. AIX

Install EN_GB UTF-8 on AIX 5.3

Hello Gang Can you please help me in installing EN_GB UTF-8 on AIX 5.3 . I have worked on Solaris and Linux, but not familier with AIX so hoping if someone show the roadmap. (5 Replies)
Discussion started by: ningy
5 Replies

9. Shell Programming and Scripting

Trying to convert utf-8 to WINDOWS-1251

Hello all i have utf-8 file that i try to convert to WINDOWS-1251 on linux without any success the file name is utf-8 when i try to do : file -bi test.txt it gives me : text/plain; charset=utf-8 when i try to convert the file i do : /usr/bin/iconv -f UTF-8 -t WINDOWS-1251 test.txt >... (1 Reply)
Discussion started by: umen
1 Replies

10. Shell Programming and Scripting

Convert UTF-8 file to ASCII/ISO8859-1 OR replace characters

I am trying to develop a script which will work on a source UTF-8 file and perform one or more of the following It will accept the target encoding as an argument e.g. US-ASCII or ISO-8859-1, etc 1. It should replace all occurrences of characters outside target character set by " " (space) or... (3 Replies)
Discussion started by: hemkiran.s
3 Replies
encoding(n)						       Tcl Built-In Commands						       encoding(n)

__________________________________________________________________________________________________________________________________________________

NAME
encoding - Manipulate encodings SYNOPSIS
encoding option ?arg arg ...? _________________________________________________________________ INTRODUCTION
Strings in Tcl are encoded using 16-bit Unicode characters. Different operating system interfaces or applications may generate strings in other encodings such as Shift-JIS. The encoding command helps to bridge the gap between Unicode and these other formats. DESCRIPTION
Performs one of several encoding related operations, depending on option. The legal options are: encoding convertfrom ?encoding? data Convert data to Unicode from the specified encoding. The characters in data are treated as binary data where the lower 8-bits of each character is taken as a single byte. The resulting sequence of bytes is treated as a string in the specified encoding. If encoding is not specified, the current system encoding is used. encoding convertto ?encoding? string Convert string from Unicode to the specified encoding. The result is a sequence of bytes that represents the converted string. Each byte is stored in the lower 8-bits of a Unicode character. If encoding is not specified, the current system encoding is used. encoding dirs ?directoryList? Tcl can load encoding data files from the file system that describe additional encodings for it to work with. This command sets the | search path for *.enc encoding data files to the list of directories directoryList. If directoryList is omitted then the command | returns the current list of directories that make up the search path. It is an error for directoryList to not be a valid list. If, | when a search for an encoding data file is happening, an element in directoryList does not refer to a readable, searchable direc- | tory, that element is ignored. encoding names Returns a list containing the names of all of the encodings that are currently available. encoding system ?encoding? Set the system encoding to encoding. If encoding is omitted then the command returns the current system encoding. The system encod- ing is used whenever Tcl passes strings to system calls. EXAMPLE
It is common practice to write script files using a text editor that produces output in the euc-jp encoding, which represents the ASCII characters as singe bytes and Japanese characters as two bytes. This makes it easy to embed literal strings that correspond to non-ASCII characters by simply typing the strings in place in the script. However, because the source command always reads files using the current system encoding, Tcl will only source such files correctly when the encoding used to write the file is the same. This tends not to be true in an internationalized setting. For example, if such a file was sourced in North America (where the ISO8859-1 is normally used), each byte in the file would be treated as a separate character that maps to the 00 page in Unicode. The resulting Tcl strings will not contain the expected Japanese characters. Instead, they will contain a sequence of Latin-1 characters that correspond to the bytes of the original string. The encoding command can be used to convert this string to the expected Japanese Unicode characters. For example, set s [encoding convertfrom euc-jp "xA4xCF"] would return the Unicode string "u306F", which is the Hiragana letter HA. SEE ALSO
Tcl_GetEncoding(3) KEYWORDS
encoding Tcl 8.1 encoding(n)
All times are GMT -4. The time now is 03:04 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy