Shell Programming and Scripting

Hello,
I have a special problem. I have a file in 8 bit and would like to convert the whole database to 16Bit unicode.
The mapping file has the following structure:

The mapper is provided as a zip file

The target file to be converted contains data in English and 8 bit Urdu mapping, a sample of which is also given below:
I have tried to use the mapper to convert but seem to get entangled in 8 bit and 16 bit conversion.

The only other way is to map to hex values and write a program in C which would be long and cumbersome
Any help given would be really great.
Many thanks in advance.

As far as i can see "sed" should be up to the task: go through your mapper file, one line at a time. There are 4 bytes in every line:

1. a "src" character
2. a ">"
3,4. a "target" character.

It is easy to divide in 2 shell variables:

Code:

cat mapper | while read line ; do
     src="${line%???}"
     tgt="${line#??}"
done

Now use these variables to replace all original characters to translated ones. You have to mark you original characters somehow to make sure you don't translate characters twice during various passes. In this case we escape all characters initially with a backslash and remove these as we translate. This way we can - by searching for backslashes in the resultfile - find out if our mapper file is completely covering the input. If still backslashes are there some characters didn't get translated.

Code:

sed 's/./\\&//g' /path/to/infile > workfile # mask all characters with "\"

cat mapper | while read line ; do
     src="${line%???}"
     tgt="${line#??}"
     sed 's/\\'"${src}"'/'"${tgt}"'/g' workfile > workfile.tmp
     mv workfile.tmp workfile
done
mv workfile /path/to/outfile

This silently presupposes the backslash "\" not to be used in your inputfile. Replace it with another (unused) character if this is not the case.

I hope this helps.

bakunin

Many thanks. I forgot to mention that I work on Windows and do not have SED. Is there a PERL or awk solution for the same ?
Many thanks once again

If you use "awk" i suppose you already have some part of the "MKS-toolkit" (or whatever it is called today, "Unix Tools for Windows" or something such - i don't use any Windows) installed. You can download and install "sed" from the same source you already got "awk" from.

I hope this helps.

bakunin

Many thanks. Will try the solution out and get back to you