plucene::analysis::chartokenizer(3pm) debian man page | unix.com

Plucene::Analysis::CharTokenizer(3pm)			User Contributed Perl Documentation		     Plucene::Analysis::CharTokenizer(3pm)

NAME
       Plucene::Analysis::CharTokenizer - base class for character tokenisers

SYNOPSIS
	       # isa Plucene::Analysis::Tokenizer

	       my $next = $chartokenizer->next;

DESCRIPTION
       This is an abstract base class for simple, character-oriented tokenizers.

METHODS
   token_re
       This should be defined in subclasses.

   normalize
       This will normalise the character before it is added to the token.

   next
	       my $next = $chartokenizer->next;

       This will return the next token in the string, or undef at the end of the string.

perl v5.12.4							    2011-08-14				     Plucene::Analysis::CharTokenizer(3pm)