Sponsored Content
Top Forums Shell Programming and Scripting Convert UTF-8 file to ASCII/ISO8859-1 OR replace characters Post 302980631 by RudiC on Wednesday 31st of August 2016 03:22:30 AM
Old 08-31-2016
That is a complex specification to read (and understand). If I did - which I don't dare to assert - a solution to replace "extended" characters with "nomal" ones would be
Code:
echo "\"#$%&0@ABab£¥§©ほぼぽま~अ" | LC_ALL=C sed 'p; s/[\xe0-\xef]../X/g; s/[\xc0-\xcf]./Y/g'
"#$%&0@ABab£¥§©ほぼぽま~अ
"#$%&0@ABabYYYYXXXX~X

Here the characters in question are replaced by X or Y, depending on their representation in UTF-8 - replace them by what you want, e.g. spaces or nothing. If you want to translate to EUR, do so BEFORE those substitutions above, but be aware that then your "fixed length" will be lost.

Last edited by RudiC; 08-31-2016 at 04:55 AM..
 

10 More Discussions You Might Find Interesting

1. Programming

Howto convert Ascii -> UTF-8 & back C++

While working with russian text under FreeBSD&MySQL I need to convert a string from MySQL to the Unicode format. I've just started my way in C++ under FreeBSD , so please explain me how can I get ascii code of Char variable and also how can i get a character into variable with the specified ascii... (3 Replies)
Discussion started by: macron
3 Replies

2. Shell Programming and Scripting

Replace characters in a string using their ascii value

Hi All, In the HP Unix that i'm using when i initialise a string as Stalled="'30¬G'" Stalled=$Stalled" '30¬C'", it is taking the character ¬ as a comma. I need to grep for 30¬G 30¬C in a file and take its count. But since this character ¬ is not being understood, the count returns a zero. The... (2 Replies)
Discussion started by: roops
2 Replies

3. Shell Programming and Scripting

replace UTF-8 characters with tr

Hi, I try to get tr to replace multibytes characters by ascii equivalent. For example "Je vais à l'école" ---> 'Je vais a l'ecole" But my version of tr (5.97) doesn't seem to support multibyte sets. $ locale charmap; echo "Je vais à l'école" | tr éà ea UTF-8 Je vais aa l'aacole I try to... (2 Replies)
Discussion started by: ripat
2 Replies

4. Shell Programming and Scripting

read in a file character by character - replace any unknown ASCII characters with spa

Can someone help me to write a script / command to read in a file, character by character, replace any unknown ASCII characters with space. then write out the file to a new filename/ Thanks! (1 Reply)
Discussion started by: raghav525
1 Replies

5. Shell Programming and Scripting

convert ascii values into ascii characters

Hi gurus, I have a file in unix with ascii values. I need to convert all the ascii values in the file to ascii characters. File contains nearly 20000 records with ascii values. (10 Replies)
Discussion started by: sandeeppvk
10 Replies

6. Shell Programming and Scripting

Remove characters other than ISO8859-1

Hi please help in writing a script for replacing all the non-iso8859-1 characters to question marks. I need a pattern of this kind "sed s/<non-iso char range>/?/g < ipfile > opfile" Please help me in this. (2 Replies)
Discussion started by: rprajendran
2 Replies

7. Red Hat

Can't convert 7bit ASCII to UTF-8

Hello, I am trying to convert a 7bit ASCII file to UTF-8. I have used iconv before though it can't recognize it for some reason and says unknown file encoding. When I used ascii2uni package with different package, ./ascii2uni -a K -a I -a J -a X test_file > new_test_file It still... (2 Replies)
Discussion started by: rockf1bull
2 Replies

8. Linux

Help to Convert file from UNIX UTF-8 to Windows UTF-16

Hi, I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt and i am getting some chinese characters as below which l opened the converted file on windows machine. LANG=en_US.UTF-8... (3 Replies)
Discussion started by: phanidhar6039
3 Replies

9. Shell Programming and Scripting

Search and Replace Extended Ascii Characters

We are getting extended Ascii characters in the input file and my requirement is to search and replace them with a space. I am using the following command LANG=C sed -e 's// /g' It is doing a good job, but in some cases it is replacing the extended characters with two spaces. So my input... (12 Replies)
Discussion started by: ysvsr1
12 Replies

10. Shell Programming and Scripting

Convert Hex to Ascii in a Ascii file

Hi All, I have an ascii file in which few columns are having hex values which i need to convert into ascii. Kindly suggest me what command can be used in unix shell scripting? Thanks in Advance (2 Replies)
Discussion started by: HemaV
2 Replies
PRECONV(1)						      General Commands Manual							PRECONV(1)

NAME
preconv - convert encoding of input files to something GNU troff understands SYNOPSIS
preconv [-dr] [-e encoding] [files ...] preconv -h | --help preconv -v | --version It is possible to have whitespace between the -e command line option and its parameter. DESCRIPTION
preconv reads files and converts its encoding(s) to a form GNU troff(1) can process, sending the data to standard output. Currently, this means ASCII characters and '[uXXXX]' entities, where 'XXXX' is a hexadecimal number with four to six digits, representing a Unicode input code. Normally, preconv should be invoked with the -k and -K options of groff. OPTIONS
-d Emit debugging messages to standard error (mainly the used encoding). -Dencoding Specify default encoding if everything fails (see below). -eencoding Specify input encoding explicitly, overriding all other methods. This corresponds to groff's -Kencoding option. Without this switch, preconv uses the algorithm described below to select the input encoding. --help -h Print help message. -r Do not add .lf requests. --version -v Print version number. USAGE
preconv tries to find the input encoding with the following algorithm. 1. If the input encoding has been explicitly specified with option -e, use it. 2. Otherwise, check whether the input starts with a Byte Order Mark (BOM, see below). If found, use it. 3. Finally, check whether there is a known coding tag (see below) in either the first or second input line. If found, use it. 4. If everything fails, use a default encoding as given with option -D, by the current locale, or 'latin1' if the locale is set to 'C', 'POSIX', or empty (in that order). Note that the groff program supports a GROFF_ENCODING environment variable which is eventually expanded to option -k. Byte Order Mark The Unicode Standard defines character U+FEFF as the Byte Order Mark (BOM). On the other hand, value U+FFFE is guaranteed not be a Unicode character at all. This allows to detect the byte order within the data stream (either big-endian or lower-endian), and the MIME encodings 'UTF-16' and 'UTF-32' mandate that the data stream starts with U+FEFF. Similarly, the data stream encoded as 'UTF-8' might start with a BOM (to ease the conversion from and to UTF-16 and UTF-32). In all cases, the byte order mark is not part of the data but part of the encoding protocol; in other words, preconv's output doesn't contain it. Note that U+FEFF not at the start of the input data actually is emitted; it has then the meaning of a 'zero width no-break space' character - something not needed normally in groff. Coding Tags Editors which support more than a single character encoding need tags within the input files to mark the file's encoding. While it is pos- sible to guess the right input encoding with the help of heuristic algorithms for data which represents a greater amount of a natural lan- guage, it is still just a guess. Additionally, all algorithms fail easily for input which is either too short or doesn't represent a natu- ral language. For these reasons, preconv supports the coding tag convention (with some restrictions) as used by GNU Emacs and XEmacs (and probably other programs too). Coding tags in GNU Emacs and XEmacs are stored in so-called File Variables. preconv recognizes the following syntax form which must be put into a troff comment in the first or second line. -*- tag1: value1; tag2: value2; ... -*- The only relevant tag for preconv is 'coding' which can take the values listed below. Here an example line which tells Emacs to edit a file in troff mode, and to use latin2 as its encoding. ." -*- mode: troff; coding: latin-2 -*- The following list gives all MIME coding tags (either lowercase or uppercase) supported by preconv; this list is hard-coded in the source. big5, cp1047, euc-jp, euc-kr, gb2312, iso-8859-1, iso-8859-2, iso-8859-5, iso-8859-7, iso-8859-9, iso-8859-13, iso-8859-15, koi8-r, us-ascii, utf-8, utf-16, utf-16be, utf-16le In addition, the following hard-coded list of other tags is recognized which eventually map to values from the list above. ascii, chinese-big5, chinese-euc, chinese-iso-8bit, cn-big5, cn-gb, cn-gb-2312, cp878, csascii, csisolatin1, cyrillic-iso-8bit, cyrillic-koi8, euc-china, euc-cn, euc-japan, euc-japan-1990, euc-korea, greek-iso-8bit, iso-10646/utf8, iso-10646/utf-8, iso-latin-1, iso-latin-2, iso-latin-5, iso-latin-7, iso-latin-9, japanese-euc, japanese-iso-8bit, jis8, koi8, korean-euc, korean-iso-8bit, latin-0, latin1, latin-1, latin-2, latin-5, latin-7, latin-9, mule-utf-8, mule-utf-16, mule-utf-16be, mule-utf-16-be, mule-utf-16be-with-signature, mule-utf-16le, mule-utf-16-le, mule-utf-16le-with-signature, utf8, utf-16-be, utf-16-be-with-signature, utf-16be-with-signature, utf-16-le, utf-16-le-with-signature, utf-16le-with-signature Those tags are taken from GNU Emacs and XEmacs, together with some aliases. Trailing '-dos', '-unix', and '-mac' suffixes of coding tags (which give the end-of-line convention used in the file) are stripped off before the comparison with the above tags happens. Iconv Issues preconv by itself only supports three encodings: latin-1, cp1047, and UTF-8; all other encodings are passed to the iconv library functions. At compile time it is searched and checked for a valid iconv implementation; a call to 'preconv --version' shows whether iconv is used. BUGS
preconv doesn't support local variable lists yet. This is a different syntax form to specify local variables at the end of a file. SEE ALSO
groff(1) the GNU Emacs and XEmacs info pages COPYING
Copyright (C) 2006-2014 Free Software Foundation, Inc. Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that this permission notice may be included in translations approved by the Free Software Foundation instead of in the original English. Groff Version 1.22.3 10 February 2018 PRECONV(1)
All times are GMT -4. The time now is 02:26 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy