Convert UTF-8 file to ASCII/ISO8859-1 OR replace characters
I am trying to develop a script which will work on a source UTF-8 file and perform one or more of the following
It will accept the target encoding as an argument e.g. US-ASCII or ISO-8859-1, etc
1. It should replace all occurrences of characters outside target character set by " " (space) or whatever character we define. Naturally we don't want to shift character positions in case of fixed width files.
2. It should also have the ability to get rid of characters altogether, characters which fall outside target character set.
3. It should have the ability to translate certain characters .e.g. "¥" to "JPY" or "£" to "GBP" or "Â" to "A"
What I have tried
1. This is good to get rid of characters, but it doesn't work for fixed width. Further it has //TRANSLIT option but one has no control over character translation.
2. This gives me a list of characters > \xff
Trying to replace them with a space or any other character.
Even sed -e option doesn't work here, same error.
Using the hexdump
If I plug in this value in sed this works.
So my final approach is parse the file and use grep to find characters outside the range of target encoding and put only matching characters into a temp file with their hexdump values
Then loop through temp file and for each character do the character replacement or suppress characters one by one.
I am stuck here
1. Is there a way to define hexdump values of characters in a variable. This will be used in association with grep command to create temp file
below does not work for me.
OR
2. Is there a way to store the hex value in some variable and use sed or anything else to perform find replace?
here find_char = each character read from temp file one by one
Even this does not work.
I have tried lot of options and googled at least 100 webpages to get some clues. I am open to ideas, suggestions...
I am using
Moderator's Comments:
Please use CODE tags as required by forum rules!
Last edited by RudiC; 08-31-2016 at 03:46 AM..
Reason: Changed CODE tags.
While working with russian text under FreeBSD&MySQL I need to convert a string from MySQL to the Unicode format.
I've just started my way in C++ under FreeBSD , so please explain me how can I get ascii code of Char variable and also how can i get a character into variable with the specified ascii... (3 Replies)
Hi All,
In the HP Unix that i'm using when i initialise a string as Stalled="'30¬G'"
Stalled=$Stalled" '30¬C'", it is taking the character ¬ as a comma. I need to grep for 30¬G 30¬C in a file and take its count. But since this character ¬ is not being understood, the count returns a zero.
The... (2 Replies)
Hi,
I try to get tr to replace multibytes characters by ascii equivalent. For example
"Je vais à l'école" ---> 'Je vais a l'ecole"
But my version of tr (5.97) doesn't seem to support multibyte sets.
$ locale charmap; echo "Je vais à l'école" | tr éà ea
UTF-8
Je vais aa l'aacole
I try to... (2 Replies)
Can someone help me to write a script / command to read in a file, character by character, replace any unknown ASCII characters with space. then write out the file to a new filename/
Thanks! (1 Reply)
Hi gurus,
I have a file in unix with ascii values. I need to convert all the ascii values in the file to ascii characters. File contains nearly 20000 records with ascii values. (10 Replies)
Hi please help in writing a script for replacing all the non-iso8859-1 characters to question marks.
I need a pattern of this kind
"sed s/<non-iso char range>/?/g < ipfile > opfile"
Please help me in this. (2 Replies)
Hello,
I am trying to convert a 7bit ASCII file to UTF-8.
I have used iconv before though it can't recognize it for some reason and says unknown file encoding.
When I used ascii2uni package with different package, ./ascii2uni -a K -a I -a J -a X test_file > new_test_file
It still... (2 Replies)
Hi,
I have tried to convert a UTF-8 file to windows UTF-16 format file as below from unix machine
unix2dos < testing.txt | iconv -f UTF-8 -t UTF-16 > out.txt
and i am getting some chinese characters as below which l opened the converted file on windows machine.
LANG=en_US.UTF-8... (3 Replies)
We are getting extended Ascii characters in the input file and my requirement is to search and replace them with a space. I am using the following command
LANG=C sed -e 's// /g'
It is doing a good job, but in some cases it is replacing the extended characters with two spaces. So my input... (12 Replies)
Hi All,
I have an ascii file in which few columns are having hex values which i need to convert into ascii. Kindly suggest me what command can be used in unix shell scripting?
Thanks in Advance (2 Replies)