I'm thinking that perhaps there is no direct or equivalent character to translate these characters to in your destination character set, and so that's why they're being dropped, maybe ?
Some testing of my own. Firstly, all I did here was copy and paste the string you provided:
and it was picked up as UTF-8, as you can see. Full disclosure: this was on a Slackware Linux 14.2 system.
So here's what happens when I try converting this to ASCII, and as mentioned I think it fails since these characters simply don't exist in any way in normal ASCII:
However, if I tell iconv to transliterate only what it can, and drop what it can't, things seem to work, although I end up with question marks in the output (since there's nothing to transliterate to):
So I think that's the issue: they're being dropped or giving errors because there isn't anything in your destination character set that iconv regards as an acceptable replacement.
Hi,
One of our application is producing log files. But if we open the log file in vi or less or view mode, it shows all the special characters in it. The 'cat' shows correctly but it shows only last page. If I do 'cat' <file_name> | more, then again it shows special characters.
... (1 Reply)
Hi,
I need some advise on treating non printable chars over ascii value 126
Case 1 :
On some fields in the text , I need to retiain then 'as-is' and load to a database.I understand it also depends on database codepage.
but i just wanna know how do i ensure it do not change while loading... (1 Reply)
here is my simple script to show process and owners except me:
ps `-ef |grep xterm |grep -v aucar` | while read a1 a2 a3 a4 a5 a6 a7 a8
do
echo KILL..\($a1\).. $a2 |more
done
how can I pass values from command "ps -ef |grep xterm|grep -v aucar" to ?
because above command... (2 Replies)
I was trying to run a code to check if a fax number is empty or not.
for that, I've written the following code which is throwing an error.
#!/bin/ksh
fax= "999-999-9999"
if ; then
fax_no="000-000-0000"
else
fax_no=$fax
fi
echo $fax_no
And I get the... (7 Replies)
Hi, I'm having trouble with awk print all characters between 2 patterns. I tried more then one solution found on this forum but with no success.
Probably my mistakes are due to the special characters "" and "]"in the search patterns.
Well, have a log file like this:
logfile.txt
... (3 Replies)
I have a file with multiple lines. From each line I want to get all strings that starts with '+' and ends with '/'. Then I want the strings to be separated by ' + '
Example input:
+$A$/NOUN+At/NSUFF_FEM_PL+K/CASE_INDEF_ACC
Sample output:
$A$ + At + K (20 Replies)
Hi guys,
I am trying to find the following string in a file, but I always get pattern not found error, not sure what is missing here. Can you help please?
I do a less to open the xrates.log and then do a /'="18"' in the file and tried various combinations to search the below string.
String... (8 Replies)
Running SunOs 5.6. Solaris.
I've been able to remove all special characters from a fixed length file which appear in the first column but as a result all subsequent columns have shifted to the left by the amount of characters deleted.
It is a space separated file. Line 1 in input file is... (6 Replies)
Hi all!!
I´m using command file -i myfile.xml to validate XML file encoding, but it is just saying regular file . I´m expecting / looking an output as UTF8 or ANSI / ASCII
Is there command to display the files encoding?
Thank you! (2 Replies)
Hi Team,
I have a file a1.txt with data as follows.
dfjakjf...asdfkasj</EnableQuotedIDs><SQL><SelectStatement modified='1' type='string'><!
The delimiter string: <SelectStatement modified='1' type='string'><!
dlm="<SelectStatement modified='1' type='string'><!
The above command is... (7 Replies)
Discussion started by: kmanivan82
7 Replies
LEARN ABOUT XFREE86
iconv
ICONV(1) Linux User Manual ICONV(1)NAME
iconv - convert text from one character encoding to another
SYNOPSIS
iconv [options] [-f from-encoding] [-t to-encoding] [inputfile]...
DESCRIPTION
The iconv program reads in text in one encoding and outputs the text in another encoding. If no input files are given, or if it is given
as a dash (-), iconv reads from standard input. If no output file is given, iconv writes to standard output.
If no from-encoding is given, the default is derived from the current locale's character encoding. If no to-encoding is given, the default
is derived from the current locale's character encoding.
OPTIONS -f from-encoding, --from-code=from-encoding
Use from-encoding for input characters.
-t to-encoding, --to-code=to-encoding
Use to-encoding for output characters.
If the string //IGNORE is appended to to-encoding, characters that cannot be converted are discarded and an error is printed after
conversion.
If the string //TRANSLIT is appended to to-encoding, characters being converted are transliterated when needed and possible. This
means that when a character cannot be represented in the target character set, it can be approximated through one or several similar
looking characters. Characters that are outside of the target character set and cannot be transliterated are replaced with a ques-
tion mark (?) in the output.
-l, --list
List all known character set encodings.
-c Silently discard characters that cannot be converted instead of terminating when encountering such characters.
-o outputfile, --output=outputfile
Use outputfile for output.
-s, --silent
This option is ignored; it is provided only for compatibility.
--verbose
Print progress information on standard error when processing multiple files.
-?, --help
Print a usage summary and exit.
--usage
Print a short usage summary and exit.
-V, --version
Print the version number, license, and disclaimer of warranty for iconv.
EXIT STATUS
Zero on success, nonzero on errors.
ENVIRONMENT
Internally, the iconv program uses the iconv(3) function which in turn uses gconv modules (dynamically loaded shared libraries) to convert
to and from a character set. Before calling iconv(3), the iconv program must first allocate a conversion descriptor using iconv_open(3).
The operation of the latter function is influenced by the setting of the GCONV_PATH environment variable:
* If GCONV_PATH is not set, iconv_open(3) loads the system gconv module configuration cache file created by iconvconfig(8) and then, based
on the configuration, loads the gconv modules needed to perform the conversion. If the system gconv module configuration cache file is
not available then the system gconv module configuration file is used.
* If GCONV_PATH is defined (as a colon-separated list of pathnames), the system gconv module configuration cache is not used. Instead,
iconv_open(3) first tries to load the configuration files by searching the directories in GCONV_PATH in order, followed by the system
default gconv module configuration file. If a directory does not contain a gconv module configuration file, any gconv modules that it
may contain are ignored. If a directory contains a gconv module configuration file and it is determined that a module needed for this
conversion is available in the directory, then the needed module is loaded from that directory, the order being such that the first
suitable module found in GCONV_PATH is used. This allows users to use custom modules and even replace system-provided modules by pro-
viding such modules in GCONV_PATH directories.
FILES
/usr/lib/gconv
Usual default gconv module path.
/usr/lib/gconv/gconv-modules
Usual system default gconv module configuration file.
/usr/lib/gconv/gconv-modules.cache
Usual system gconv module configuration cache.
CONFORMING TO
POSIX.1-2001.
EXAMPLE
Convert text from the ISO 8859-15 character encoding to UTF-8:
$ iconv -f ISO-8859-15 -t UTF-8 < input.txt > output.txt
The next example converts from UTF-8 to ASCII, transliterating when possible:
$ echo abc B a EUR ac | iconv -f UTF-8 -t ASCII//TRANSLIT
abc ss ? EUR abc
SEE ALSO locale(1), iconv(3), nl_langinfo(3), charsets(7), iconvconfig(8)COLOPHON
This page is part of release 4.15 of the Linux man-pages project. A description of the project, information about reporting bugs, and the
latest version of this page, can be found at https://www.kernel.org/doc/man-pages/.
GNU 2018-02-02 ICONV(1)