Shell Programming and Scripting

Hi all

I have a bilingual text file, one language is English and another one is Assamese. I want to write English language text into one file and Assamese language text into another file. For Example-

input file is

as dsad জন gdfkghdfkghkdf hksdjhfkjsdfhk hksjdhfksdjf আনজনৰ মনত ghgj jkj uieurtier

out put file-1

as dsad gdfkghdfkghkdf hksdjhfkjsdfhk hksjdhfksdjf ghgj jkj uieurtier

output file-2

জন আনজনৰ মনত

Please help me !!!

Is the input file unicode?

yes. Assamesse text are in Unicode {UTF-8}

That means each leading byte of Assamese text characters has the following bit pattern:

1110---- (highest order bit first) --- are bits used for data. not typing of characters.

This is the pattern for ASCII:
0------

The spaces present a problem because they exist in each character set.

You need a bit of C code to test bits, then write either one byte to an output stream
for ASCII or three bytes to an output stream opened for Assamese text. Since the spaces are common to both, I would write them as a newline and worrry about them later.

Lastly, convert the newlines in the two output files back into space characters after you are done.

I think that writing the space to the same file as the previous (non space) byte, would eliminate this problem, as converting spaces to newlines, may introduce a problem with paragraph breaks etc.
Also is punctuation common?

I was curious as to how this problem would be solved.

As jim mentioned above, all Assamese characters begin with a byte sequence that is not present in English text. So, differentiating between the two is easy enough.

From what I gathered, Assamese script uses characters that are also used in Bengali, and their Unicode range is 0x0980-0x09FF, which when encoded into UTF-8 is a range of three-byte sequences: 11100000 10100110 10000000 - 11100000 10100111 10111111 (hopefully I didn't mess that up ).

The leading byte's value in that range is always 0xe0 (0340 octal, 224 decimal). We can test for its presence with AWK:

Code:

awk '/\xe0/ {print > "assamese"; next} {print > "english"}' data

Regards,
Alister

Yes sir punctuations are common?