unihist(1) [debian man page]

unihist(1)						      General Commands Manual							unihist(1)

NAME

       unihist - Generate a histogram of the characters in a Unicode file

SYNOPSIS

       unihist ([option flags])

DESCRIPTION

       unihist	generates  a  histogram  of the characters in its input, which must be encoded in UTF-8 Unicode. By default, for each character it
       prints the frequency of the character as a percentage of the total, the absolute number of tokens in the input, the UTF-32 code	in   hexa-
       decimal,  and, if the character is displayable, the glyph itself as UTF-8 Unicode. Command line flags allow unwanted information to be sup-
       pressed.  In particular, note that by suppressing the percentages and counts it is possible to generate a list of the unique characters	in
       the input.

       Output is produced ordered by character code. To sort it in descending order of frequency, pipe the output into the command:

	      sort -k1 -n -r

       By  default,  unihist  handles all of Unicode. To reduce memory usage and increase speed, it may be compiled so as to handle only the Basic
       Multilingual Plane (plane 0) by defining BMPONLY.

COMMAND LINE FLAGS

       -c     Suppress printing of counts and percentages.

       -g     Suppress printing of glyphs.

       -h     Print usage information.

       -u     Suppress printing of the Unicode code as text.

       -v     Print version information.

SEE ALSO

       uniname (1)

REFERENCES

       Unicode Standard, version 5.0

AUTHOR

       Bill Poser
       billposer@alum.mit.edu

LICENSE

       GNU General Public License

								     May, 2008								unihist(1)

unifuzz(1)						      General Commands Manual							unifuzz(1)

NAME

       unifuzz - Emit strings designed to test Unicode handling

SYNOPSIS

       unifuzz ([option flags])

DESCRIPTION

       unifuzz	emits strings designed to test the ability of programs intended to accept Unicode input to handle unexpected input. These include:
       characters from all Unicode ranges, Private Use characters, surrogates, undefined characters, non-characters,  control  characters,  exotic
       space characters, sequences violating normalization rules, unexpected sequences (e.g. a base character from one range followed by a combin-
       ing character from another range), and long sequences of combining characters.  It can also generate very long  lines,  strings	containing
       embedded nulls, and ill-formed UTF-8.

COMMAND LINE FLAGS

       -b     Restrict the output to the Basic Multilingual Plane (Plane 0).

       -g     Do not emit specific characters.

       -h     Print usage information.

       -l     Emit very long lines.

       -n     Emit string with embedded nulls.

       -q     Be quiet. Omit commentary.

       -r <number>
	      Set the number of random characters to emit.

       -S     Scan ranges - emit a character from each range.

       -s <seed>
	      Set the seed for the random number generator.

       -u     Emit ill-formed UTF-8.

       -v     Print version information.

       The  sequence  of  random  characters is determined by a pseudorandom number generator, so the same sequence can be obtained by setting the
       seed to the same value. If not set on the command line, a seed is chosen based on the time of execution. The seed used is included  in  the
       output  in  a  line  of the form "Seed = NNNNNN" immediately preceding the random character sequence. Note that in order to obtain the same
       sequence it is necessary to keep the same setting for restriction of output to the BMP.

REFERENCES

       Unicode Standard, version 5.0

AUTHOR

       Bill Poser
       billposer@alum.mit.edu

LICENSE

       GNU General Public License

								    April, 2008 							unifuzz(1)