Distributed Lucene : A distributed free text index for Hadoop


 
Thread Tools Search this Thread
Special Forums News, Links, Events and Announcements UNIX and Linux RSS News Distributed Lucene : A distributed free text index for Hadoop
# 1  
Old 06-09-2008
Distributed Lucene : A distributed free text index for Hadoop

HPL-2008-64 Distributed Lucene : A distributed free text index for Hadoop - Butler, Mark H.; Rutherford, James
Keyword(s): distributed, high availability, free text, parallel, search
Abstract: This technical report described a parallel, distributed free text index written at HP Labs called Distributed Lucene. Distributed Lucene is based on two Apache open source projects, Lucene and Hadoop. It was written to gain a better understanding of the Apache Hadoop architecture, which is derived f ...
Full Report

More...
Login or Register to Ask a Question

Previous Thread | Next Thread

5 More Discussions You Might Find Interesting

1. Filesystems, Disks and Memory

distributed filesystem over internet/VPN

On this forum was already posted similar question, but it was 4 years ago and didn't give me answers. I have two groups of engineers that works in far locations connected via VPN. Physically, the connection is a DSL. Currently we have a linux server in one location that provide files over... (4 Replies)
Discussion started by: Domino
4 Replies

2. Programming

Distributed Automation kill()

Ok, so Ive been stumped by this problem for the last 3 days, and my exploration is reaching its limitations... The key idea is that at any moment in time, the automation controller could theoretically have any number of threads open to any number of node machines running any number of... (2 Replies)
Discussion started by: jjinno
2 Replies

3. High Performance Computing

Grid vs. Parallel vs. Distributed

Hello all, I was wondering if someone could either explain or maybe point me to another article somewhere that explains the difference between: distributed computing grid computing parallel computing I see these terms thrown around a lot in server and cluster environments, but I'd like a... (2 Replies)
Discussion started by: Heathe_Kyle
2 Replies

4. Programming

Creating a Unique ID on distributed systems

Hi, How do you actually create a unique ID on a distributed system. I looked at gethostid but the man page says that its not guaranteed to be unique. Also using the IP address does not seem to be a feasible solution. Is there a function call or mechanism by which this is possible when even the... (4 Replies)
Discussion started by: pic
4 Replies

5. UNIX for Dummies Questions & Answers

Distributed

I've downloaded distributed latest build for dnetc, and I think i installed it. Not sure where the excicuteable would be. Any help? i realize this is a little ambigious, so I can allways reintstall it if i know how to put it in a specific directory.. does anyone know if VNC works for the Intel... (3 Replies)
Discussion started by: veitcha
3 Replies
Login or Register to Ask a Question
Plucene(3pm)						User Contributed Perl Documentation					      Plucene(3pm)

NAME
Plucene - A Perl port of the Lucene search engine SYNOPSIS
Create Documents by adding Fields: my $doc = Plucene::Document->new; $doc->add(Plucene::Document::Field->Text(content => $content)); $doc->add(Plucene::Document::Field->Text(author => "Your Name")); Choose Your Analyser and add documents to an Index Writer my $analyzer = Plucene::Analysis::SimpleAnalyzer->new(); my $writer = Plucene::Index::Writer->new("my_index", $analyzer, 1); $writer->add_document($doc); undef $writer; # close Search by building a Query my $parser = Plucene::QueryParser->new({ analyzer => Plucene::Analysis::SimpleAnalyzer->new(), default => "text" # Default field for non-specified queries }); my $query = $parser->parse('author:"Your Name"'); Then pass the Query to an IndexSearcher and collect hits my $searcher = Plucene::Search::IndexSearcher->new("my_index"); my @docs; my $hc = Plucene::Search::HitCollector->new(collect => sub { my ($self, $doc, $score) = @_; push @docs, $searcher->doc($doc); }); $searcher->search_hc($query => $hc); DESCRIPTION
Plucene is a fully-featured and highly customizable search engine toolkit based on the Lucene API. (<http://jakarta.apache.org/lucene>) It is not, in and of itself, a functional search engine - you are expected to subclass and tie all the pieces together to suit your own needs. The synopsis above gives a rough indication of how to use the engine in simple cases. See Plucene::Simple for one example of tying it all together. The tests shipped with Plucene provide a variety of other examples of how use this. EXTENSIONS
Plucene comes shipped with some default Analyzers. However it is expected that users will want to create Analyzers to meet their own needs. To avoid namespace corruption, anyone releasing such Analyzers to CPAN (which is encouraged!) should place them in the namespace Plucene::Plugin::Analyzer::. DOCUMENTATION
Although most of the Perl modules should be well documented, the Perl API mirrors Lucene's to such an extent that reading Lucene's documentation will give you a good idea of how to do more advanced stuff with Plucene. See particularly the ONJava articles <http://www.onjava.com/pub/a/onjava/2003/01/15/lucene.html> and <http://www.onjava.com/pub/a/onjava/2003/03/05/lucene.html>. These are brilliant introductions to the concepts surrounding Lucene, how it works, and how to extend it. COMPATIBILITY
For the most part Lucene and Plucene indexes are created in the same manner. However, due to current implementation details, the indexes will generally not be compatible. It should theoretically be possible to convert index files in either direction between Plucene and Lucene, but no tools are currently provided to do so. As Plucene is still undergoing development, we cannot guarantee index format compatibility across releases. If you're using Plucene in production code, you need to ensure that you can recreate the indexes. MISSING FEATURES
The following features have not yet been fully implemented: o Wildcard searches o Range searches MAILING LIST
Bug reports, patches, queries, discussion etc should be addressed to the mailing list. More information on the list can be found at: <http://www.kasei.com/mailman/listinfo/plucene> AUTHORS
Initially ported by Simon Cozens and Marc Kerr. Currently maintained by Tony Bowden and Marty Pauley. Original Java Lucene by Doug Cutting and others. THANKS
The initial development and ongoing maintenance of Plucene has been funded and supported by Kasei <http://www.kasei.com/> LICENSE
This software is licensed under the same terms as Perl itself. perl v5.12.4 2011-08-14 Plucene(3pm)