unihist(1) [debian man page]
unihist(1) General Commands Manual unihist(1) NAME
unihist - Generate a histogram of the characters in a Unicode file SYNOPSIS
unihist ([option flags]) DESCRIPTION
unihist generates a histogram of the characters in its input, which must be encoded in UTF-8 Unicode. By default, for each character it prints the frequency of the character as a percentage of the total, the absolute number of tokens in the input, the UTF-32 code in hexa- decimal, and, if the character is displayable, the glyph itself as UTF-8 Unicode. Command line flags allow unwanted information to be sup- pressed. In particular, note that by suppressing the percentages and counts it is possible to generate a list of the unique characters in the input. Output is produced ordered by character code. To sort it in descending order of frequency, pipe the output into the command: sort -k1 -n -r By default, unihist handles all of Unicode. To reduce memory usage and increase speed, it may be compiled so as to handle only the Basic Multilingual Plane (plane 0) by defining BMPONLY. COMMAND LINE FLAGS
-c Suppress printing of counts and percentages. -g Suppress printing of glyphs. -h Print usage information. -u Suppress printing of the Unicode code as text. -v Print version information. SEE ALSO
uniname (1) REFERENCES
Unicode Standard, version 5.0 AUTHOR
Bill Poser billposer@alum.mit.edu LICENSE
GNU General Public License May, 2008 unihist(1)
Check Out this Related Man Page
UTF(6) Games Manual UTF(6) NAME
UTF, Unicode, ASCII, rune - character set and format DESCRIPTION
The Plan 9 character set and representation are based on the Unicode Standard and on the ISO multibyte UTF-8 encoding (Universal Character Set Transformation Format, 8 bits wide). The Unicode Standard represents its characters in 16 bits; UTF-8 represents such values in an 8-bit byte stream. Throughout this manual, UTF-8 is shortened to UTF. In Plan 9, a rune is a 16-bit quantity representing a Unicode character. Internally, programs may store characters as runes. However, any external manifestation of textual information, in files or at the interface between programs, uses a machine-independent, byte-stream encoding called UTF. UTF is designed so the 7-bit ASCII set (values hexadecimal 00 to 7F), appear only as themselves in the encoding. Runes with values above 7F appear as sequences of two or more bytes with values only from 80 to FF. The UTF encoding of the Unicode Standard is backward compatible with ASCII: programs presented only with ASCII work on Plan 9 even if not written to deal with UTF, as do programs that deal with uninterpreted byte streams. However, programs that perform semantic processing on ASCII graphic characters must convert from UTF to runes in order to work properly with non-ASCII input. See rune(2). Letting numbers be binary, a rune x is converted to a multibyte UTF sequence as follows: 01. x in [00000000.0bbbbbbb] -> 0bbbbbbb 10. x in [00000bbb.bbbbbbbb] -> 110bbbbb, 10bbbbbb 11. x in [bbbbbbbb.bbbbbbbb] -> 1110bbbb, 10bbbbbb, 10bbbbbb Conversion 01 provides a one-byte sequence that spans the ASCII character set in a compatible way. Conversions 10 and 11 represent higher- valued characters as sequences of two or three bytes with the high bit set. Plan 9 does not support the 4, 5, and 6 byte sequences pro- posed by X-Open. When there are multiple ways to encode a value, for example rune 0, the shortest encoding is used. In the inverse mapping, any sequence except those described above is incorrect and is converted to rune hexadecimal 0080. FILES
/lib/unicode table of characters and descriptions, suitable for look(1). SEE ALSO
ascii(1), tcs(1), rune(2), keyboard(6), The Unicode Standard. UTF(6)