Test::LectroTest::TestRunner(3pm)			User Contributed Perl Documentation			 Test::LectroTest::TestRunner(3pm)

NAME

       Test::LectroTest::TestRunner - Configurable TAP-compatible engine for running LectroTest property checks

SYNOPSIS

	use Test::LectroTest::TestRunner;

	my @args = trials => 1_000, retries => 20_000;
	my $runner = Test::LectroTest::TestRunner->new( @args );

	# test a single property and print details upon failure
	my $result = $runner->run( $a_single_lectrotest_property );
	print $result->details unless $result->success;

	# test a suite of properties, w/ Test::Harness::TAP output
	my $num_successful = $runner->run_suite( @properties );
	print "# All passed!" if $num_successful == @properties;

DESCRIPTION

       STOP! If you just want to write and run simple tests, see Test::LectroTest.  If you really want to learn about the property-checking
       apparatus or turn its control knobs, read on.

       This module provides Test::LectroTest::TestRunner, a class of objects that tests properties by running repeated random trials.  Create a
       TestRunner, configure it, and then call its "run" or "run_suite" methods to test properties individually or in groups.

METHODS

       The following methods are available.

   new(named-params)
	 my $runner = new Test::LectroTest::TestRunner(
	   trials      => 1_000,
	   retries     => 20_000,
	   scalefn     => sub { $_[0] / 2 + 1 },
	   verbose     => 1,
	   regressions => "/path/to/regression_suite.txt",
	 );

       Creates a new Test::LectroTest::TestRunner and configures it with the given named parameters, if any.  Typically, you need only provide the
       "trials" parameter because the other values are reasonable for almost all situations.  Here is what each parameter means:

       trials
	   The number of trials to run against each property checked.  The default is 1_000.

       retries
	   The number of times to allow a property to retry trials (via "$tcon->retry") during the entire property check before aborting the
	   check.  This is used to prevent infinite looping, should the property retry every attempt.

       scalefn
	   A subroutine that scales the sizing guidance given to input generators.

	   The TestRunner starts with an initial guidance of 1 at the beginning of a property check.  For each trial (or retry) of the property,
	   the guidance value is incremented.  This causes successive trials to be tried using successively more complex inputs.  The "scalefn"
	   subroutine gets to adjust this guidance on the way to the input generators.	Typically, you would change the "scalefn" subroutine if
	   you wanted to change the rate and which inputs grow during the course of the trials.

       verbose
	   If this parameter is set to true (the default) the TestRunner will use verbose output that includes things like label frequencies and
	   counterexamples.  Otherwise, only one-line summaries will be output.  Unless you have a good reason to do otherwise, leave this
	   parameter alone because verbose output is almost always what you want.

       record_failures
	   If this parameter is set to a file's pathname (or a FailureRecorder object), the TestRunner will record property-check failures to the
	   file (or recorder).	(This is an easy way to build a regression-testing suite.)  If the file cannot be created or written to, this
	   parameter will be ignored.  Set this parameter to "undef" (the default) to turn off recording.

       playback_failures
	   If this parameter is set to a file's pathname (or a FailureRecorder object), the TestRunner will load previously recorded failures from
	   the file (or recorder) and use them as additional test cases when checking properties.  If the file cannot be read, this option will be
	   ignored.  Set this parameter to "undef" (the default) to turn off recording.

       regressions
	   If this parameter is set to a file's pathname (or a FailureRecorder object), the TestRunner will load failures from and record failures
	   to the file (or recorder).  Setting this parameter is a shortcut for, and exactly equivalent to, setting record_failures and
	   <playback_failures> to the same value, which is typically what you want when managing a persistent suite of regression tests.

	   This is a write-only accessor.

       You can also set and get the values of the configuration properties using accessors of the same name.  For example:

	 $runner->trials( 10_000 );

   run(property)
	 $results = $runner->run( $a_property );
	 print $results->summary, "
";
	 if ($results->success) {
	     # celebrate!
	 }

       Checks whether the given property holds by running repeated random trials.  The result is a Test::LectroTest::TestRunner::results object,
       which you can query for fined-grained information about the outcome of the check.

       The "run" method takes an optional second argument which gives the test number.	If it is not provided (usually the case), the next number
       available from the TestRunner's internal counter is used.

	 $results = $runner->run( $third_property, 3 );

       Additionally, if the TestRunner's playback_failures parameter is defined, this method will play back any relevant failure cases from the
       given playback file (or FailureRecorder).

       Additionally, if the TestRunner's record_failures parameter is defined, this method will record any new failures to the given file (or
       FailureRecorder).

   run_suite(properties...)
	 my $num_successful = $runner->run_suite( @properties );
	 if ($num_successful == @properties) {
	     # celebrate most jubilantly!
	 }

       Checks a suite of properties, sending the results of each property checked to "STDOUT" in a form that is compatible with
       Test::Harness::TAP.  For example:

	 1..5
	 ok 1 - Property->new disallows use of 'tcon' in bindings
	 ok 2 - magic Property syntax disallows use of 'tcon' in bindings
	 ok 3 - exceptions are caught and reported as failures
	 ok 4 - pre-flight check catches new w/ no args
	 ok 5 - pre-flight check catches unbalanced arguments list

       By default, labeling statistics and counterexamples (if any) are included in the output if the TestRunner's "verbose" property is true.
       You may override the default by passing the "verbose" named parameter after all of the properties in the argument list:

	 my $num_successes = $runner->run_suite( @properties,
						 verbose => 1 );
	 my $num_failed = @properties - $num_successes;

HELPER OBJECTS

       There are two kinds of objects that TestRunner uses as helpers.	Neither is meant to be created by you.	Rather, a TestRunner will create
       them on your behalf when they are needed.

       The objects are described in the following subsections.

   Test::LectroTest::TestRunner::results
	 my $results = $runner->run( $a_property );
	 print "Property name: ", $results->name, ": ";
	 print $results->success ? "Winner!" : "Loser!";

       This is the object that you get back from "run".  It contains all of the information available about the outcome of a property check and
       provides the following methods:

       success
	   Boolean value:  True if the property checked out successfully; false otherwise.

       summary
	   Returns a one line summary of the property-check outcome.  It does not end with a newline.  Example:

	     ok 1 - Property->new disallows use of 'tcon' in bindings

       details
	   Returns all relevant information about the property-check outcome as a series of lines.  The last line is terminated with a newline.
	   The details are identical to the summary (except for the terminating newline) unless label frequencies are present or a counterexample
	   is present, in which case the details will have these extras (the summary does not).  Example:

	     1..1
	     not ok 1 - 'my_sqrt meets defn of sqrt' falsified in 1 attempts
	     # Counterexample:
	     # $x = '0.546384454460178';

       name
	   Returns the name of the property to which the results pertain.

       number
	   The number assigned to the property that was checked.

       counterexample
	   Returns the counterexample that "broke" the code being tested, if there is one.  Otherwise, returns an empty string.  If any notes have
	   been attached to the failing trial, they will be included.

       labels
	   Label counts.  If any labels were applied to trials during the property check, this value will be a reference to a hash mapping each
	   combination of labels to the count of trials that had that particular combination.  Otherwise, it will be undefined.

	   Note that each trial is counted only once -- for the most-specific combination of labels that was applied to it.  For example, consider
	   the following labeling logic:

	     Property {
	       ##[ x <- Int ]##
	       $tcon->label("negative") if $x < 0;
	       $tcon->label("odd")	if $x % 2;
	       1;
	     }, name => "negative/odd labeling example";

	   For a particular trial, if x was 2 (positive and even), the trial would receive no labels.  If x was 3 (positive and odd), the trial
	   would be labeled "odd".  If x was -2 (negative and even), the trial would be labeled "negative".  If x was -3 (negative and odd), the
	   trial would be labeled "negative & odd".

       label_frequencies
	   Returns a string containing a line-by-line accounting of labels applied during the series of trials:

	     print $results->label_frequencies;

	   The corresponding output looks like this:

	     25% negative
	     25% negative & odd
	     25% odd

	   If no labels were applied, an empty string is returned.

       exception
	   Returns the text of the exception or error that caused the series of trials to be aborted, if the trials were aborted because an
	   exception or error was intercepted by LectroTest.  Otherwise, returns an empty string.

       attempts
	   Returns the count of trials performed.

       incomplete
	   In the event that the series of trials was halted before it was completed (such as when the retry count was exhausted), this method
	   will return the reason.  Otherwise, it returns an empty string.

	   Note that a series of trials is complete if a counterexample was found.

   Test::LectroTest::TestRunner::testcontroller
       During a live property-check trial, the variable $tcon is available to your Properties.	It lets you label the current trial or request
       that it be re-tried with new inputs.

       The following methods are available.

       retry
	       Property {
		 ##[ x <- Int ]##
		 return $tcon->retry if $x == 0;
	       }, ... ;

	   Stops the current trial and tells the TestRunner to re-try it with new inputs.  Typically used to reject a particular case of inputs
	   that doesn't make for a good or valid test.	While not required, you will probably want to call "$tcon->retry" as part of a "return"
	   statement to prevent further execution of your property's logic, the results of which will be thrown out should it run to completion.

	   The return value of "$tcon->retry" is itself meaningless; it is the side-effect of calling it that causes the current trial to be
	   thrown out and re-tried.

       label(string)
	       Property {
		 ##[ x <- Int ]##
		 $tcon->label("negative") if $x < 0;
		 $tcon->label("odd")	  if $x % 2;
	       }, ... ;

	   Applies a label to the current trial.  At the end of the trial, all of the labels are gathered together, and the trial is dropped into
	   a bucket bearing the combined label.  See the discussion of "labels" for more.

       trivial
	       Property {
		 ##[ x <- Int ]##
		 $tcon->trivial if $x == 0;
	       }, ... ;

	   Applies the label "trivial" to the current trial.  It is identical to calling "label" with "trivial" as the argument.

       note(string...)
	       Property {
		 ##[ s <- String( charset=>"A-Za-z0-9" ) ]##
		 my $s_enc     = encode($s);
		 my $s_enc_dec = decode($s_enc);
		 $tcon->note("s_enc	= $s_enc",
			     "s_enc_dec = $s_enc_dec");
		 $s eq $s_enc_dec;
	       }, name => "decode is encode's inverse" ;

	   Adds a note (or notes) to the current trial.  In the event that the trial fails, these notes will be emitted as part of the
	   counterexample.  For example:

	       1..1
	       not ok 1 - property 'decode is encode's inverse' 
		   falsified in 68 attempts
	       #     Counterexample:
	       #     $s = "0";
	       #     Notes:
	       #     $s_enc	= "";
	       #     $s_enc_dec = "";

	   Notes can help you debug your code when something goes wrong.  Use them as debugging hints to yourself.  For example, you can use notes
	   to record the output of each stage of a multi-stage test.  That way, if the test fails, you can see what happened in each stage without
	   having to plug the counterexample into your code under a debugger.

	   If you want to include complicated values or data structures in your notes, see the "dump" method, next, which may be more appropriate.

       dump(value, name)
	       Property {
		 ##[ s <- String ]##
		 my $s_enc     = encode($s);
		 my $s_enc_dec = decode($s_enc);
		 $tcon->dump($s_enc, "s_enc");
		 $tcon->dump($s_enc_dec, "s_enc_dec");
		 $s eq $s_enc_dec;
	       }, name => "decode is encode's inverse" ;

	   Adds a note to the current trial in which the given value is dumped.  The value will be dumped via Data::Dumper and thus may be complex
	   and contain weird control characters and so on.  If you supply a name, it will be used to name the dumped value.  Returns value as its
	   result.

	   In the event that the trial fails, the note (and any others) will be emitted as part of the counterexample.

	   See "note" above for more.

SEE ALSO

       Test::LectroTest::Property explains in detail what you can put inside of your property specifications.

       Test::LectroTest::RegressionTesting explains how to test for regressions and corner cases using LectroTest.

       Test::Harness:TAP documents the Test Anything Protocol, Perl's simple text-based interface between testing modules such as Test::LectroTest
       and the test harness Test::Harness.

LECTROTEST HOME

       The LectroTest home is http://community.moertel.com/LectroTest.	There you will find more documentation, presentations, mailing-list
       archives, a wiki, and other helpful LectroTest-related resources.  It's also the best place to ask questions.

AUTHOR

       Tom Moertel (tom@moertel.com)

INSPIRATION

       The LectroTest project was inspired by Haskell's QuickCheck module by Koen Claessen and John Hughes:
       http://www.cs.chalmers.se/~rjmh/QuickCheck/.

COPYRIGHT and LICENSE
       Copyright (c) 2004-06 by Thomas G Moertel.  All rights reserved.

       This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.12.3							    2011-05-12					 Test::LectroTest::TestRunner(3pm)