scraper

Web::Scraper(3pm)					User Contributed Perl Documentation					 Web::Scraper(3pm)

NAME

       Web::Scraper - Web Scraping Toolkit using HTML and CSS Selectors or XPath expressions

SYNOPSIS

	 use URI;
	 use Web::Scraper;

	 # First, create your scraper block
	 my $tweets = scraper {
	     # Parse all LIs with the class "status", store them into a resulting
	     # array 'tweets'.	We embed another scraper for each tweet.
	     process "li.status", "tweets[]" => scraper {
		 # And, in that array, pull in the elementy with the class
		 # "entry-content", "entry-date" and the link
		 process ".entry-content", body => 'TEXT';
		 process ".entry-date", when => 'TEXT';
		 process 'a[rel="bookmark"]', link => '@href';
	     };
	 };

	 my $res = $tweets->scrape( URI->new("http://twitter.com/miyagawa") );

	 # The result has the populated tweets array
	 for my $tweet (@{$res->{tweets}}) {
	     print "$tweet->{body} $tweet->{when} (link: $tweet->{link})
";
	 }

       The structure would resemble this (visually)
	 {
	   tweets => [
	     { body => $body, when => $date, link => $uri },
	     { body => $body, when => $date, link => $uri },
	   ]
	 }

DESCRIPTION

       Web::Scraper is a web scraper toolkit, inspired by Ruby's equivalent Scrapi. It provides a DSL-ish interface for traversing HTML documents
       and returning a neatly arranged Perl data strcuture.

       The scraper and process blocks provide a method to define what segments of a document to extract.  It understands HTML and CSS Selectors as
       well as XPath expressions.

METHODS

   scraper
	 $scraper = scraper { ... };

       Creates a new Web::Scraper object by wrapping the DSL code that will be fired when scrape method is called.

   scrape
	 $res = $scraper->scrape(URI->new($uri));
	 $res = $scraper->scrape($html_content);
	 $res = $scraper->scrape($html_content);
	 $res = $scraper->scrape($http_response);
	 $res = $scraper->scrape($html_element);

       Retrieves the HTML from URI, HTTP::Response, HTML::Tree or text strings and creates a DOM object, then fires the callback scraper code to
       retrieve the data structure.

       If you pass URI or HTTP::Response object, Web::Scraper will automatically guesses the encoding of the content by looking at Content-Type
       headers and META tags. Otherwise you need to decode the HTML to Unicode before passing it to scrape method.

       You can optionally pass the base URL when you pass the HTML content as a string instead of URI or HTTP::Response.

	 $res = $scraper->scrape($html_content, "http://example.com/foo");

       This way Web::Scraper can resolve the relative links found in the document.

   process
	 scraper {
	     process "tag.class", key => 'TEXT';
	     process '//tag[contains(@foo, "bar")]', key2 => '@attr';
	     process '//comment()', 'comments[]' => 'TEXT';
	 };

       process is the method to find matching elements from HTML with CSS selector or XPath expression, then extract text or attributes into the
       result stash.

       If the first argument begins with "//" or "id(" it's treated as an XPath expression and otherwise CSS selector.

	 # <span class="date">2008/12/21</span>
	 # date => "2008/12/21"
	 process ".date", date => 'TEXT';

	 # <div class="body"><a href="http://example.com/">foo</a></div>
	 # link => URI->new("http://example.com/")
	 process ".body > a", link => '@href';

	 # <div class="body"><!-- HTML Comment here --><a href="http://example.com/">foo</a></div>
	 # comment => " HTML Comment here "
	 #
	 # NOTES: A comment nodes are accessed when installed
	 # the HTML::TreeBuilder::XPath (version >= 0.14) and/or
	 # the HTML::TreeBuilder::LibXML (version >= 0.13)
	 process "//div[contains(@class, 'body')]/comment()", comment => 'TEXT';

	 # <div class="body"><a href="http://example.com/">foo</a></div>
	 # link => URI->new("http://example.com/"), text => "foo"
	 process ".body > a", link => '@href', text => 'TEXT';

	 # <ul><li>foo</li><li>bar</li></ul>
	 # list => [ "foo", "bar" ]
	 process "li", "list[]" => "TEXT";

	 # <ul><li id="1">foo</li><li id="2">bar</li></ul>
	 # list => [ { id => "1", text => "foo" }, { id => "2", text => "bar" } ];
	 process "li", "list[]" => { id => '@id', text => "TEXT" };

EXAMPLES

       There are many examples in the "eg/" dir packaged in this distribution.	It is recommended to look through these.

NESTED SCRAPERS

       TBD

FILTERS

       TBD

AUTHOR

       Tatsuhiko Miyagawa <miyagawa@bulknews.net>

LICENSE

       This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

SEE ALSO

       <http://blog.labnotes.org/category/scrapi/>

       HTML::TreeBuilder::XPath

perl v5.14.2							    2011-11-19							 Web::Scraper(3pm)