MKDoc::XML::Tokenizer(3pm)				User Contributed Perl Documentation				MKDoc::XML::Tokenizer(3pm)

NAME

       MKDoc::XML::Tokenizer - Tokenize XML the REX way

SYNOPSIS

	 my $tokens = MKDoc::XML::Tokenizer->process_data ($some_xml);
	 foreach my $token (@{$tokens})
	 {
	     print "'" . $token->as_string() . "' is text
" if (defined $token->text());
	     print "'" . $token->as_string() . "' is a self closing tag
" if (defined $token->tag_self_close());
	     print "'" . $token->as_string() . "' is an opening tag
" if (defined $token->tag_open());
	     print "'" . $token->as_string() . "' is a closing tag
" if (defined $token->tag_close());
	     print "'" . $token->as_string() . "' is a processing instruction
" if (defined $token->pi());
	     print "'" . $token->as_string() . "' is a declaration
" if (defined $token->declaration());
	     print "'" . $token->as_string() . "' is a comment
" if (defined $token->comment());
	     print "'" . $token->as_string() . "' is a tag
" if (defined $token->tag());
	     print "'" . $token->as_string() . "' is a pseudo-tag (NOT text and NOT tag)
" if (defined $token->pseudotag());
	     print "'" . $token->as_string() . "' is a leaf token (NOT opening tag)
" if (defined $token->leaf());
	 }

SUMMARY

       MKDoc::XML::Tokenizer is a module which uses Robert D. Cameron REX technique to parse XML (ignore the carriage returns):

	 [^<]+|<(?:!(?:--(?:[^-]*-(?:[^-][^-]*-)*->?)?|[CDATA[(?:[^]]*](?:[^]]+])
	 *]+(?:[^]>][^]]*](?:[^]]+])*]+)*>)?|DOCTYPE(?:[ 
	
]+(?:[A-Za-z_:]|[^
	 x00-x7F])(?:[A-Za-z0-9_:.-]|[^x00-x7F])*(?:[ 
	
]+(?:(?:[A-Za-z_:]|[^
	 x00-x7F])(?:[A-Za-z0-9_:.-]|[^x00-x7F])*|"[^"]*"|'[^']*'))*(?:[ 
	
]+)
	 ?(?:[(?:<(?:!(?:--[^-]*-(?:[^-][^-]*-)*->|[^-](?:[^]"'><]+|"[^"]*"|'[^']*'
	 )*>)|?(?:[A-Za-z_:]|[^x00-x7F])(?:[A-Za-z0-9_:.-]|[^x00-x7F])*(?:?>|[
	 n
	 ][^?]*?+(?:[^>?][^?]*?+)*>))|%(?:[A-Za-z_:]|[^x00-x7F])(?:[A-Za-z0
	 -9_:.-]|[^x00-x7F])*;|[ 
	
]+)*](?:[ 
	
]+)?)?>?)?)?|?(?:(?:[A-Za-z
	 _:]|[^x00-x7F])(?:[A-Za-z0-9_:.-]|[^x00-x7F])*(?:?>|[

	 ][^?]*?+(?
	 :[^>?][^?]*?+)*>)?)?|/(?:(?:[A-Za-z_:]|[^x00-x7F])(?:[A-Za-z0-9_:.-]|[^x
	 00-x7F])*(?:[ 
	
]+)?>?)?|(?:(?:[A-Za-z_:]|[^x00-x7F])(?:[A-Za-z0-9_:.
	 -]|[^x00-x7F])*(?:[ 
	
]+(?:[A-Za-z_:]|[^x00-x7F])(?:[A-Za-z0-9_:.-]|
	 [^x00-x7F])*(?:[ 
	
]+)?=(?:[ 
	
]+)?(?:"[^<"]*"|'[^<']*'))*(?:[ 

	 t
]+)?/?>?)?)

       That's right. One big regex, and it works rather well.

DISCLAIMER

       This module does low level XML manipulation. It will somehow parse even broken XML and try to do something with it. Do not use it unless
       you know what you're doing.

API

   my $tokens = MKDoc::XML::Tokenizer->process_data ($some_xml);
       Splits $some_xml into a list of MKDoc::XML::Token objects and returns an array reference to the list of tokens.

   my $tokens = MKDoc::XML::Tokenizer->process_file ('/some/file.xml');
       Same as MKDoc::XML::Tokenizer->process_data ($some_xml), except that it reads $some_xml from '/some/file.xml'.

NOTES

       MKDoc::XML::Tokenizer works with MKDoc::XML::Token, which can be used when building a full tree is not necessary. If you need to build a
       tree, look at MKDoc::XML::TreeBuilder.

AUTHOR

       Copyright 2003 - MKDoc Holdings Ltd.

       Author: Jean-Michel Hiver

       This module is free software and is distributed under the same license as Perl itself. Use it at your own risk.

SEE ALSO

       MKDoc::XML::Token MKDoc::XML::TreeBuilder

perl v5.10.1							    2004-10-06						MKDoc::XML::Tokenizer(3pm)