html::linkextor(3) [mojave man page]
HTML::LinkExtor(3) User Contributed Perl Documentation HTML::LinkExtor(3) NAME
HTML::LinkExtor - Extract links from an HTML document SYNOPSIS
require HTML::LinkExtor; $p = HTML::LinkExtor->new(&cb, "http://www.perl.org/"); sub cb { my($tag, %links) = @_; print "$tag @{[%links]} "; } $p->parse_file("index.html"); DESCRIPTION
HTML::LinkExtor is an HTML parser that extracts links from an HTML document. The HTML::LinkExtor is a subclass of HTML::Parser. This means that the document should be given to the parser by calling the $p->parse() or $p->parse_file() methods. $p = HTML::LinkExtor->new $p = HTML::LinkExtor->new( $callback ) $p = HTML::LinkExtor->new( $callback, $base ) The constructor takes two optional arguments. The first is a reference to a callback routine. It will be called as links are found. If a callback is not provided, then links are just accumulated internally and can be retrieved by calling the $p->links() method. The $base argument is an optional base URL used to absolutize all URLs found. You need to have the URI module installed if you provide $base. The callback is called with the lowercase tag name as first argument, and then all link attributes as separate key/value pairs. All non-link attributes are removed. $p->links Returns a list of all links found in the document. The returned values will be anonymous arrays with the following elements: [$tag, $attr => $url1, $attr2 => $url2,...] The $p->links method will also truncate the internal link list. This means that if the method is called twice without any parsing between them the second call will return an empty list. Also note that $p->links will always be empty if a callback routine was provided when the HTML::LinkExtor was created. EXAMPLE
This is an example showing how you can extract links from a document received using LWP: use LWP::UserAgent; use HTML::LinkExtor; use URI::URL; $url = "http://www.perl.org/"; # for instance $ua = LWP::UserAgent->new; # Set up a callback that collect image links my @imgs = (); sub callback { my($tag, %attr) = @_; return if $tag ne 'img'; # we only look closer at <img ...> push(@imgs, values %attr); } # Make the parser. Unfortunately, we don't know the base yet # (it might be different from $url) $p = HTML::LinkExtor->new(&callback); # Request document and parse it as it arrives $res = $ua->request(HTTP::Request->new(GET => $url), sub {$p->parse($_[0])}); # Expand all image URLs to absolute ones my $base = $res->base; @imgs = map { $_ = url($_, $base)->abs; } @imgs; # Print them out print join(" ", @imgs), " "; SEE ALSO
HTML::Parser, HTML::Tagset, LWP, URI::URL COPYRIGHT
Copyright 1996-2001 Gisle Aas. This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.18.2 2013-03-25 HTML::LinkExtor(3)
Check Out this Related Man Page
Tagset(3) User Contributed Perl Documentation Tagset(3) NAME
HTML::Tagset - data tables useful in parsing HTML SYNOPSIS
use HTML::Tagset; # Then use any of the items in the HTML::Tagset package # as need arises DESCRIPTION
This module contains several data tables useful in various kinds of HTML parsing operations. Note that all tag names used are lowercase. In the following documentation, a "hashset" is a hash being used as a set -- the hash conveys that its keys are there, and the actual val- ues associated with the keys are not significant. (But what values are there, are always true.) hashset %HTML::Tagset::emptyElement This hashset has as values the tag-names (GIs) of elements that cannot have content. (For example, "base", "br", "hr".) So $HTML::Tagset::emptyElement{'hr'} exists and is true. $HTML::Tagset::emptyElement{'dl'} does not exist, and so is not true. hashset %HTML::Tagset::optionalEndTag This hashset lists tag-names for elements that can have content, but whose end-tags are generally, "safely", omissible. Example: $HTML::Tagset::emptyElement{'li'} exists and is true. hash %HTML::Tagset::linkElements Values in this hash are tagnames for elements that might contain links, and the value for each is a reference to an array of the names of attributes whose values can be links. hash %HTML::Tagset::boolean_attr This hash (not hashset) lists what attributes of what elements can be printed without showing the value (for example, the "noshade" attribute of "hr" elements). For elements with only one such attribute, its value is simply that attribute name. For elements with many such attributes, the value is a reference to a hashset containing all such attributes. hashset %HTML::Tagset::isPhraseMarkup This hashset contains all phrasal-level elements. hashset %HTML::Tagset::is_Possible_Strict_P_Content This hashset contains all phrasal-level elements that be content of a P element, for a strict model of HTML. hashset %HTML::Tagset::isHeadElement This hashset contains all elements that elements that should be present only in the 'head' element of an HTML document. hashset %HTML::Tagset::isList This hashset contains all elements that can contain "li" elements. hashset %HTML::Tagset::isTableElement This hashset contains all elements that are to be found only in/under a "table" element. hashset %HTML::Tagset::isFormElement This hashset contains all elements that are to be found only in/under a "form" element. hashset %HTML::Tagset::isBodyMarkup This hashset contains all elements that are to be found only in/under the "body" element of an HTML document. hashset %HTML::Tagset::isHeadOrBodyElement This hashset includes all elements that I notice can fall either in the head or in the body. hashset %HTML::Tagset::isKnown This hashset lists all known HTML elements. hashset %HTML::Tagset::canTighten This hashset lists elements that might have ignorable whitespace as children or siblings. array @HTML::Tagset::p_closure_barriers This array has a meaning that I have only seen a need for in "HTML::TreeBuilder", but I include it here on the off chance that someone might find it of use: When we see a "<p>" token, we go lookup up the lineage for a p element we might have to minimize. At first sight, we might say that if there's a p anywhere in the lineage of this new p, it should be closed. But that's wrong. Consider this document: <html> <head> <title>foo</title> </head> <body> <p>foo <table> <tr> <td> foo <p>bar </td> </tr> </table> </p> </body> </html> The second p is quite legally inside a much higher p. My formalization of the reason why this is legal, but this: <p>foo<p>bar</p></p> isn't, is that something about the table constitutes a "barrier" to the application of the rule about what p must minimize. So @HTML::Tagset::p_closure_barriers is the list of all such barrier-tags. hashset %isCDATA_Parent This hashset includes all elements whose content is CDATA. CAVEATS
You may find it useful to alter the behavior of modules (like "HTML::Element" or "HTML::TreeBuilder") that use "HTML::Tagset"'s data tables by altering the data tables themselves. You are welcome to try, but be careful; and be aware that different modules may or may react dif- ferently to the data tables being changed. Note that it may be inappropriate to use these tables for producing HTML -- for example, %isHeadOrBodyElement lists the tagnames for all elements that can appear either in the head or in the body, such as "script". That doesn't mean that I am saying your code that produces HTML should feel free to put script elements in either place! If you are producing programs that spit out HTML, you should be intimately familiar with the DTDs for HTML or XHTML (available at "http://www.w3.org/"), and you should slavishly obey them, not the data tables in this document. SEE ALSO
HTML::Element, HTML::TreeBuilder, HTML::LinkExtor COPYRIGHT
Copyright 1995-2000 Gisle Aas; copyright 2000 Sean M. Burke. This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. AUTHOR
Current maintainer: Sean M. Burke, <sburke@cpan.org> Most of the code/data in this module was adapted from code written by Gisle Aas <gisle@aas.no> for "HTML::Element", "HTML::TreeBuilder", and "HTML::LinkExtor". perl v5.8.0 2000-10-20 Tagset(3)