Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Change encoding, no removing special chars. inconv Post 303011132 by drysdalk on Saturday 13th of January 2018 05:25:08 PM
Old 01-13-2018
Hi,

I'm thinking that perhaps there is no direct or equivalent character to translate these characters to in your destination character set, and so that's why they're being dropped, maybe ?

Some testing of my own. Firstly, all I did here was copy and paste the string you provided:

Code:
$ cat test
í, ó,
$ file test
test: UTF-8 Unicode text
$

and it was picked up as UTF-8, as you can see. Full disclosure: this was on a Slackware Linux 14.2 system.

So here's what happens when I try converting this to ASCII, and as mentioned I think it fails since these characters simply don't exist in any way in normal ASCII:

Code:
$ iconv -f=utf8 -t=ascii -o new.txt test.txt
iconv: illegal input sequence at position 0
$

However, if I tell iconv to transliterate only what it can, and drop what it can't, things seem to work, although I end up with question marks in the output (since there's nothing to transliterate to):

Code:
$ iconv -f=utf8 -t=ascii//TRANSLIT -o new.txt test.txt
$ cat new.txt
?, ?,
$

So I think that's the issue: they're being dropped or giving errors because there isn't anything in your destination character set that iconv regards as an acceptable replacement.

Hope this helps.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Supress special chars in vi

Hi, One of our application is producing log files. But if we open the log file in vi or less or view mode, it shows all the special characters in it. The 'cat' shows correctly but it shows only last page. If I do 'cat' <file_name> | more, then again it shows special characters. ... (1 Reply)
Discussion started by: divakarp
1 Replies

2. Shell Programming and Scripting

treating special chars

Hi, I need some advise on treating non printable chars over ascii value 126 Case 1 : On some fields in the text , I need to retiain then 'as-is' and load to a database.I understand it also depends on database codepage. but i just wanna know how do i ensure it do not change while loading... (1 Reply)
Discussion started by: braindrain
1 Replies

3. Shell Programming and Scripting

special chars arrangement in code

here is my simple script to show process and owners except me: ps `-ef |grep xterm |grep -v aucar` | while read a1 a2 a3 a4 a5 a6 a7 a8 do echo KILL..\($a1\).. $a2 |more done how can I pass values from command "ps -ef |grep xterm|grep -v aucar" to ? because above command... (2 Replies)
Discussion started by: xramm
2 Replies

4. UNIX for Dummies Questions & Answers

Strings with Special chars in IF condition

I was trying to run a code to check if a fax number is empty or not. for that, I've written the following code which is throwing an error. #!/bin/ksh fax= "999-999-9999" if ; then fax_no="000-000-0000" else fax_no=$fax fi echo $fax_no And I get the... (7 Replies)
Discussion started by: hooaamai
7 Replies

5. Shell Programming and Scripting

print all between patterns with special chars

Hi, I'm having trouble with awk print all characters between 2 patterns. I tried more then one solution found on this forum but with no success. Probably my mistakes are due to the special characters "" and "]"in the search patterns. Well, have a log file like this: logfile.txt ... (3 Replies)
Discussion started by: ginolatino
3 Replies

6. Shell Programming and Scripting

All strings within two special chars

I have a file with multiple lines. From each line I want to get all strings that starts with '+' and ends with '/'. Then I want the strings to be separated by ' + ' Example input: +$A$/NOUN+At/NSUFF_FEM_PL+K/CASE_INDEF_ACC Sample output: $A$ + At + K (20 Replies)
Discussion started by: Viernes
20 Replies

7. UNIX for Dummies Questions & Answers

How to search for a string with special chars?

Hi guys, I am trying to find the following string in a file, but I always get pattern not found error, not sure what is missing here. Can you help please? I do a less to open the xrates.log and then do a /'="18"' in the file and tried various combinations to search the below string. String... (8 Replies)
Discussion started by: santokal
8 Replies

8. UNIX for Advanced & Expert Users

Removing special chars from file and maintain field separator

Running SunOs 5.6. Solaris. I've been able to remove all special characters from a fixed length file which appear in the first column but as a result all subsequent columns have shifted to the left by the amount of characters deleted. It is a space separated file. Line 1 in input file is... (6 Replies)
Discussion started by: iffy290
6 Replies

9. Solaris

View file encoding then change encoding.

Hi all!! I´m using command file -i myfile.xml to validate XML file encoding, but it is just saying regular file . I´m expecting / looking an output as UTF8 or ANSI / ASCII Is there command to display the files encoding? Thank you! (2 Replies)
Discussion started by: mrreds
2 Replies

10. UNIX for Beginners Questions & Answers

Shell script to split data with a delimiter having chars and special chars

Hi Team, I have a file a1.txt with data as follows. dfjakjf...asdfkasj</EnableQuotedIDs><SQL><SelectStatement modified='1' type='string'><! The delimiter string: <SelectStatement modified='1' type='string'><! dlm="<SelectStatement modified='1' type='string'><! The above command is... (7 Replies)
Discussion started by: kmanivan82
7 Replies
ICONV(1)                                                         Linux User Manual                                                        ICONV(1)

NAME
iconv - convert text from one character encoding to another SYNOPSIS
iconv [options] [-f from-encoding] [-t to-encoding] [inputfile]... DESCRIPTION
The iconv program reads in text in one encoding and outputs the text in another encoding. If no input files are given, or if it is given as a dash (-), iconv reads from standard input. If no output file is given, iconv writes to standard output. If no from-encoding is given, the default is derived from the current locale's character encoding. If no to-encoding is given, the default is derived from the current locale's character encoding. OPTIONS
-f from-encoding, --from-code=from-encoding Use from-encoding for input characters. -t to-encoding, --to-code=to-encoding Use to-encoding for output characters. If the string //IGNORE is appended to to-encoding, characters that cannot be converted are discarded and an error is printed after conversion. If the string //TRANSLIT is appended to to-encoding, characters being converted are transliterated when needed and possible. This means that when a character cannot be represented in the target character set, it can be approximated through one or several similar looking characters. Characters that are outside of the target character set and cannot be transliterated are replaced with a ques- tion mark (?) in the output. -l, --list List all known character set encodings. -c Silently discard characters that cannot be converted instead of terminating when encountering such characters. -o outputfile, --output=outputfile Use outputfile for output. -s, --silent This option is ignored; it is provided only for compatibility. --verbose Print progress information on standard error when processing multiple files. -?, --help Print a usage summary and exit. --usage Print a short usage summary and exit. -V, --version Print the version number, license, and disclaimer of warranty for iconv. EXIT STATUS
Zero on success, nonzero on errors. ENVIRONMENT
Internally, the iconv program uses the iconv(3) function which in turn uses gconv modules (dynamically loaded shared libraries) to convert to and from a character set. Before calling iconv(3), the iconv program must first allocate a conversion descriptor using iconv_open(3). The operation of the latter function is influenced by the setting of the GCONV_PATH environment variable: * If GCONV_PATH is not set, iconv_open(3) loads the system gconv module configuration cache file created by iconvconfig(8) and then, based on the configuration, loads the gconv modules needed to perform the conversion. If the system gconv module configuration cache file is not available then the system gconv module configuration file is used. * If GCONV_PATH is defined (as a colon-separated list of pathnames), the system gconv module configuration cache is not used. Instead, iconv_open(3) first tries to load the configuration files by searching the directories in GCONV_PATH in order, followed by the system default gconv module configuration file. If a directory does not contain a gconv module configuration file, any gconv modules that it may contain are ignored. If a directory contains a gconv module configuration file and it is determined that a module needed for this conversion is available in the directory, then the needed module is loaded from that directory, the order being such that the first suitable module found in GCONV_PATH is used. This allows users to use custom modules and even replace system-provided modules by pro- viding such modules in GCONV_PATH directories. FILES
/usr/lib/gconv Usual default gconv module path. /usr/lib/gconv/gconv-modules Usual system default gconv module configuration file. /usr/lib/gconv/gconv-modules.cache Usual system gconv module configuration cache. CONFORMING TO
POSIX.1-2001. EXAMPLE
Convert text from the ISO 8859-15 character encoding to UTF-8: $ iconv -f ISO-8859-15 -t UTF-8 < input.txt > output.txt The next example converts from UTF-8 to ASCII, transliterating when possible: $ echo abc B a EUR ac | iconv -f UTF-8 -t ASCII//TRANSLIT abc ss ? EUR abc SEE ALSO
locale(1), iconv(3), nl_langinfo(3), charsets(7), iconvconfig(8) COLOPHON
This page is part of release 4.15 of the Linux man-pages project. A description of the project, information about reporting bugs, and the latest version of this page, can be found at https://www.kernel.org/doc/man-pages/. GNU 2018-02-02 ICONV(1)
All times are GMT -4. The time now is 08:29 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy