kinosearch1::analysis::tokenizer(3pm) [debian man page]

KinoSearch1::Analysis::Tokenizer(3pm)			User Contributed Perl Documentation		     KinoSearch1::Analysis::Tokenizer(3pm)

NAME

       KinoSearch1::Analysis::Tokenizer - customizable tokenizing

SYNOPSIS

	   my $whitespace_tokenizer
	       = KinoSearch1::Analysis::Tokenizer->new( token_re => qr/S+/, );

	   # or...
	   my $word_char_tokenizer
	       = KinoSearch1::Analysis::Tokenizer->new( token_re => qr/w+/, );

	   # or...
	   my $apostrophising_tokenizer = KinoSearch1::Analysis::Tokenizer->new;

	   # then... once you have a tokenizer, put it into a PolyAnalyzer
	   my $polyanalyzer = KinoSearch1::Analysis::PolyAnalyzer->new(
	       analyzers => [ $lc_normalizer, $word_char_tokenizer, $stemmer ], );

DESCRIPTION

       Generically, "tokenizing" is a process of breaking up a string into an array of "tokens".

	   # before:
	   my $string = "three blind mice";

	   # after:
	   @tokens = qw( three blind mice );

       KinoSearch1::Analysis::Tokenizer decides where it should break up the text based on the value of "token_re".

	   # before:
	   my $string = "Eats, Shoots and Leaves.";

	   # tokenized by $whitespace_tokenizer
	   @tokens = qw( Eats, Shoots and Leaves. );

	   # tokenized by $word_char_tokenizer
	   @tokens = qw( Eats Shoots and Leaves   );

METHODS

   new
	   # match "O'Henry" as well as "Henry" and "it's" as well as "it"
	   my $token_re = qr/
		   	     # start with a word boundary
		   w+	     # Match word chars.
		   (?:	     # Group, but don't capture...
		      'w+   # ... an apostrophe plus word chars.
		   )?	     # Matching the apostrophe group is optional.
		   	     # end with a word boundary
	       /xsm;
	   my $tokenizer = KinoSearch1::Analysis::Tokenizer->new(
	       token_re => $token_re, # default: what you see above
	   );

       Constructor.  Takes one hash style parameter.

       o   token_re - must be a pre-compiled regular expression matching one token.

COPYRIGHT

       Copyright 2005-2010 Marvin Humphrey

LICENSE, DISCLAIMER, BUGS, etc.
       See KinoSearch1 version 1.00.

perl v5.14.2							    2011-11-15				     KinoSearch1::Analysis::Tokenizer(3pm)

KinoSearch1::Searcher(3pm)				User Contributed Perl Documentation				KinoSearch1::Searcher(3pm)

NAME

       KinoSearch1::Searcher - execute searches

SYNOPSIS

	   my $analyzer = KinoSearch1::Analysis::PolyAnalyzer->new(
	       language => 'en',
	   );

	   my $searcher = KinoSearch1::Searcher->new(
	       invindex => $invindex,
	       analyzer => $analyzer,
	   );
	   my $hits = $searcher->search( query => 'foo bar' );

DESCRIPTION

       Use the Searcher class to perform queries against an invindex.

METHODS

   new
	   my $searcher = KinoSearch1::Searcher->new(
	       invindex => $invindex,
	       analyzer => $analyzer,
	   );

       Constructor.  Takes two labeled parameters, both of which are required.

       o   invindex - can be either a path to an invindex, or a KinoSearch1::Store::InvIndex object.

       o   analyzer - An object which subclasses KinoSearch1::Analysis::Analyer, such as a PolyAnalyzer.  This must be identical to the Analyzer
	   used at index-time, or the results won't match up.

   search
	   my $hits = $searcher->search(
	       query  => $query,  # required
	       filter => $filter, # default: undef (no filtering)
	   );

       Process a search and return a Hits object.  search() expects labeled hash-style parameters.

       o   query - Can be either an object which subclasses KinoSearch1::Search::Query, or a query string.  If it's a query string, it will be
	   parsed using a QueryParser and a search will be performed against all indexed fields in the invindex.  For more sophisticated
	   searching, supply Query objects, such as TermQuery and BooleanQuery.

       o   filter - Must be a KinoSearch1::Search::QueryFilter.  Search results will be limited to only those documents which pass through the
	   filter.

Caching a Searcher
       When a Searcher is created, a small portion of the invindex is loaded into memory.  For large document collections, this startup time may
       become noticeable, in which case reusing the searcher is likely to speed up your search application.  Caching a Searcher is especially
       helpful when running a high-activity app under mod_perl.

       Searcher objects always represent a snapshot of an invindex as it existed when the Searcher was created.  If you want the search results to
       reflect modifications to an invindex, you must create a new Searcher after the update process completes.

COPYRIGHT

       Copyright 2005-2010 Marvin Humphrey

LICENSE, DISCLAIMER, BUGS, etc.
       See KinoSearch1 version 1.00.

perl v5.14.2							    2011-11-15						KinoSearch1::Searcher(3pm)