As jim mentioned above, all Assamese characters begin with a byte sequence that is not present in English text. So, differentiating between the two is easy enough.
From what I gathered, Assamese script uses characters that are also used in Bengali, and their Unicode range is 0x0980-0x09FF, which when encoded into UTF-8 is a range of three-byte sequences: 11100000 10100110 10000000 - 11100000 10100111 10111111 (hopefully I didn't mess that up ).
The leading byte's value in that range is always 0xe0 (0340 octal, 224 decimal). We can test for its presence with AWK:
Code:
awk '/\xe0/ {print > "assamese"; next} {print > "english"}' data
Hi,
I have my input as follows :
I have given two entries-
From system Mon Aug 1 23:52:47 2005
Source !100000006!:
Impact !100000005!: High
Status ! 7!: New
Last Name+!100000001!:
First Name+ !100000003!:
... (4 Replies)
I have a text file with lot of rows like..
Action & Adventure|2012: Supernova NR|2009-11-01 00:01:00|2010-05-01 23:59:00|Active|3
Action & Adventure|50 Dead Men Walking|2010-01-05 00:01:00|2010-06-30 23:59:00|Active|3
Action & Adventure|Afterwards|2009-11-26 00:01:00|2010-03-26... (3 Replies)
Hi All,
I am new to this forumn as well to the UNIX, I have basic knowledge of UNIX which I studied some years ago, now I have to do some shell scripting to load data into Oracle database using sqlldr utility, whcih I am able to do. I have a requirement where I need to do following operation.
I... (10 Replies)
Hi Friends,
Could you guys help me out of this problem... I need to send an email to all the users and the email has to be picked from the text file.
text file contains the no. of records like:
Code:
giridhar
224285
847333
giridhar276@gmail.com
ramana
84849
33884... (0 Replies)
I would like to print the output beginning with a header from a seperate file like this:
awk 'BEGIN{FS="_";print ((getline < "header.txt")>0)} { if (! ($0 ~ /EL/ ) print }" input.txtWhat am i doing wrong? (4 Replies)
Hello, I have a pretty simple question, but I am new to Python and am trying to write a simple program. Put simply, I want to take a text file that looks like this:
11111 22222
33333 44444
55555 66666
77777 88888
and produce two lists, one containing the contents of the left column, one the... (0 Replies)
Hello,
I want to grep a log ("server.log") for words in a separate file ("white-list.txt") and generate a separate log file containing each line that uses a word from the "white-list.txt" file.
Putting that in bullet points:
Search through "server.log" for lines that contain any word... (15 Replies)
performing this code to read from file and print each character in separate line
works well with ASCII encoded text
void
preprocess_file (FILE *fp)
{
int cc;
for (;;)
{ cc = getc (fp);
if (cc == EOF)
break;
printf ("%c\n", cc);
}
}
int
main(int... (1 Reply)
hi,
I'm trying to print out the contents of a folder into a .txt file.
The code I'm trying amongst variations is:
ls -1 > filenames.txt
but it prints them all on the same line ie.
image102.bmpimage103.bmpimage104.bmpimage105.bmpimage106.bmp
how can I change this?
Please... (2 Replies)
Hi,
Could anyone help me with this please.
Input file --
ant 1 2 3 4
2 3 4 56 7
dog 8 9 56
ant 2 3 4 5
cvh 6 7 8
ant 1 3 45
78 0 -
Would like to split the file as soon as it encounters the word "ant" very first time.
First Output file-- ant 1 2 3 4
2 3 4 56 7
dog 8 9 56
... (2 Replies)
Discussion started by: Indra2011
2 Replies
LEARN ABOUT OSX
gb18030
GB18030(5) BSD File Formats Manual GB18030(5)NAME
gb18030 -- GB 18030 encoding method for Chinese text
SYNOPSIS
ENCODING "GB18030"
DESCRIPTION
The GB18030 encoding implements GB 18030-2000, a PRC national standard for the encoding of Chinese characters. It is a superset of the older
GB 2312-1980 and GBK encodings, and incorporates Unicode's Unihan Extension A completely. It also provides code space for all Unicode 3.0
code points.
Multibyte characters in the GB18030 encoding can be one byte, two bytes, or four bytes long. There are a total of over 1.5 million code
positions.
GB 11383-1981 (ASCII) characters are represented by single bytes in the range 0x00 to 0x7F.
Chinese characters are represented as either two bytes or four bytes. Characters that are represented by two bytes begin with a byte in the
range 0x81-0xFE and end with a byte either in the range 0x40-0x7E or 0x80-0xFE.
Characters that are represented by four bytes begin with a byte in the range 0x81-0xFE, have a second byte in the range 0x30-0x39, a third
byte in the range 0x81-0xFE and a fourth byte in the range 0x30-0x39.
SEE ALSO euc(5), gb2312(5), gbk(5), utf8(5)
Chinese National Standard GB 18030-2000: Information Technology -- Chinese ideograms coded character set for information interchange --
Extension for the basic set, March 2000.
The Unicode Standard, Version 3.0, The Unicode Consortium, 2000.
STANDARDS
The GB18030 encoding is believed to be compatible with GB 18030-2000.
BSD August 10, 2003 BSD
04-21-2010
).