debian plucene::analysis::standard::standardtokenizer man page on unix.com

Debian Man Pages All Man Pages Latest Topics Forum Categories

Query: plucene::analysis::standard::standardtokenizer

OS: debian

Section: 3pm

Format: Original Unix Latex Style Formatted with HTML and a Horizontal Scroll Bar

Plucene::Analysis::Standard::StandardTokenizer(3pm)	User Contributed Perl Documentation    Plucene::Analysis::Standard::StandardTokenizer(3pm)

NAME
       Plucene::Analysis::Standard::StandardTokenizer - standard tokenizer

SYNOPSIS
	       # isa Plucene::Analysis::CharTokenizer

DESCRIPTION
       This is the standard tokenizer.

       This should be a good tokenizer for most European-language documents.

METHODS
   token_re
       The regular expression for tokenising.

   normalize
       Remove 's and .

perl v5.12.4							    2011-08-14		       Plucene::Analysis::Standard::StandardTokenizer(3pm)

debian man page for plucene::analysis::standard::standardtokenizer