Shell Programming and Scripting

I am trying to develop a script which will work on a source UTF-8 file and perform one or more of the following
It will accept the target encoding as an argument e.g. US-ASCII or ISO-8859-1, etc
1. It should replace all occurrences of characters outside target character set by " " (space) or whatever character we define. Naturally we don't want to shift character positions in case of fixed width files.
2. It should also have the ability to get rid of characters altogether, characters which fall outside target character set.
3. It should have the ability to translate certain characters .e.g. "¥" to "JPY" or "£" to "GBP" or "Â" to "A"

What I have tried
1. This is good to get rid of characters, but it doesn't work for fixed width. Further it has //TRANSLIT option but one has no control over character translation.

Code:

 iconv -f UTF-8 -t US-ASCII//IGNORE -c utf8_file.txt

2. This gives me a list of characters > \xff

Code:

 echo "!\"#$%&0@ABab£¥§©ほぼぽま~अ" | grep -P -o '[^\x00-\xff]'
ほ
ぼ
ぽ
ま
अ

Trying to replace them with a space or any other character.

Code:

 echo "!\"#$%&0@ABab£¥§©ほぼぽま~अ" | sed 's/[^\x00-\xff]/ /g'

Error: "sed: -e expression #1, char 18: Invalid collation character"

Even sed -e option doesn't work here, same error.

Using the hexdump

Code:

 echo -n अ | hexdump -ve '1/1 " %.2x"' 
Output: e0 a4 85

If I plug in this value in sed this works.

Code:

 echo "!\"#$%&0@ABab£¥§©ほぼぽま~अ" | sed 's/\xe0\xa4\x85/" "/g'
Output: !"#$%&0@ABab£¥§©ほぼぽま~" "

So my final approach is parse the file and use grep to find characters outside the range of target encoding and put only matching characters into a temp file with their hexdump values

Code:

grep -P -o '[^\x00-\x7f]' utf_sample.txt -> temp file

Then loop through temp file and for each character do the character replacement or suppress characters one by one.

I am stuck here
1. Is there a way to define hexdump values of characters in a variable. This will be used in association with grep command to create temp file
below does not work for me.

Code:

hex_range=$'\x00-\xff'
echo "!\"#$%&0@ABab£¥§©ほぼぽま~अ" | grep -P -o '[^$(hex_range)]'

OR

Code:

 echo "!\"#$%&0@ABab£¥§©ほぼぽま~अ" | grep -P -o '[^`eval $hex_range`]'

2. Is there a way to store the hex value in some variable and use sed or anything else to perform find replace?
here find_char = each character read from temp file one by one
Even this does not work.

Code:

 find_char=`echo -n अ | hexdump -ve '1/1 " %.2x"' | sed 's/ /\\\x/g'`
echo "!\"#$%&0@ABab£¥§©ほぼぽま~अ" | sed 's/$find_char/ /g'

I have tried lot of options and googled at least 100 webpages to get some clues. I am open to ideas, suggestions...

I am using

Code:

$ uname -a
Linux <server_name> 2.6.32-573.26.1.el6.x86_64 #1 SMP Tue Apr 12 01:47:01 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
$ locale
LANG=C
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

Moderator's Comments:

Please use CODE tags as required by forum rules!

Everyone at the UNIX and Linux Forums gives their best effort to reply to all questions in a timely manner. For this reason, posting questions with subjects like "Urgent!" or "Emergency" or "Help Me!" and demanding a fast reply are not permitted in the regular forums.

For members who want a higher visibility to their questions, we suggest you post in the Emergency UNIX and Linux Support Forum. This forum is given a higher priority than our regular forums.

Posting a new question in the Emergency UNIX and Linux Support Forum requires forum Bits. We monitor this forum to help people with emergencies, but we do not not guarantee response time or best answers. However, we will treat your post with a higher priority and give our best efforts to help you.

If you have posted a question in the regular forum with a subject "Urgent" "Emergency" or similar idea, we will, more-than-likely, close your thread and post this reply, redirecting you to the proper forum.

Of course, you can always post a descriptive subject text, remove words like "Urgent" etc. (from your subject and post) and post in the regular forums at any time.


Thank you.

The UNIX and Linux Forums

Neo thatks for pointing out. I did go over the forum rules but looks like I overlooked. Btw, I am trying to edit the subject with no success.

That is a complex specification to read (and understand). If I did - which I don't dare to assert - a solution to replace "extended" characters with "nomal" ones would be

Code:

echo "\"#$%&0@ABab£¥§©ほぼぽま~अ" | LC_ALL=C sed 'p; s/[\xe0-\xef]../X/g; s/[\xc0-\xcf]./Y/g'
"#$%&0@ABab£¥§©ほぼぽま~अ
"#$%&0@ABabYYYYXXXX~X

Here the characters in question are replaced by X or Y, depending on their representation in UTF-8 - replace them by what you want, e.g. spaces or nothing. If you want to translate € to EUR, do so BEFORE those substitutions above, but be aware that then your "fixed length" will be lost.