faltest(1)							Falcon User Manuals							faltest(1)

NAME

       faltest - Falcon unit test interface.

SYNOPSIS

       faltest [-d unit_test_dir] [options] [unit_list] module_file.fam

DESCRIPTION

       The faltest command line tool is a powerful interface to the Falcon unit testing system.

       The  basic  working  principle  of  faltest is that of taking all the .fal script files contained in a directory, compile and execute them,
       eventually keeping track of errors, elapsed times and execution failures. After running all the scripts, faltest may print a report on what
       happened if requested to do so.

       A  list of one or more unit test may be indicated in the faltest command line after the options. Also, the executed tests can be limited to
       named subsets.

       The unit test directory is added to the module load path, so load directives will be resolved searching the required scripts  in  the  test
       path.

UNIT TEST SCRIPTS

       Scripts being part of unit test have to start with a common header indicating some information about them. The header is actually a format-
       ted Falcon comment which is read by the faltest utility.

       This is a typical header:

       /**********************************************
       * Falcon test suite
       *
       * ID: 10a
       * Category: cat-name
       * Subcategory: subcat-name
       * Short: The short name
       * Description:
       *   Long description of this test
       *   Spanning on many lines...
       * [/Description]
       **********************************************/

       The header has a free form; faltest recognizes the following fields, being inside a comment and eventually preceded by a "*".

       ID:
	      The only mandatory field, it declares the ID under which this unit test is known. It will be  used  in  listing  the  tests  and	in
	      selecting  them as argument of the faltest command line. The id must be an integer number, eventually followed by a single lowercase
	      letter. Similar tests should be filed under the same ID with a different specification letter.

	      Scripts not having this field will be ignored by faltest.

       Category:
	      The name of the category under which this test is filed.	Faltest can select a subset of scripts to be executed to a  certain  cate-
	      gory.

       Subcategory:
	      The name of the subcategory under which this test is filed.  Faltest can select a subset of scripts to be executed to a certain sub-
	      category.

       Short:
	      Short description (or symbolic name) for this unit test.

       Description:
	      A longer description explaining what this test is supposed to do. The description always spans on more lines, and  is  closed  by  a
	      [/Description] tag.

THE TESTSUITE MODULE

       Falcon  system  provides  a  loadable  module  called  "testsuite". The module is actually embedded in faltest , and is provided to all the
       scripts it runs. The module provides the following functions:

       success()
	      The script is terminated and recorded as a success.

       failure( reason )
	      The script is terminated and recorded as a failure. An optional parameter containing a description of the failure condition  may	be
	      optionally provided; it will be written as part of the report and may be used to track which part of the test wasn't working.

       testReflect( item )
	      Returns the passed item. This is used to test for engine responsiveness to external calls and item copy through external functions.

       alive( percent )
	      In  tests  running  for some human sensible time, this function should be called periodically to inform the system and the user that
	      the test is proceeding.

	      An optional "percent" parameter can be provided. It will be interpreted as a value between 1 and 100 representing the amount of test
	      that has been performed up to this moment.

       timings( total_time, performed_ops )
	      In  case	the  execution time is relevant for the test, like in benchmarks, this function can be used to communicate back to faltest
	      the time elapsed in the operations being tested and the number of operations performed. Those parameters will be recorded and  even-
	      tually saved in the report file, to be used as benchmarks against falcon engine modifications.

       timeFactor()
	      Lengthy  tests  are  often performed by looping over the operation to be tested for a certain time. Benchmarks and performance tests
	      should be written so that they can normally complete in a reasonable time, between one and ten seconds. In case the user	wants  the
	      tests  to  perform  longer,  in  order to obtain better statistical data, it can pass the -f (time factor) option to faltest command
	      line. The time factor will be a number greater than 1, and should be used by tests that may perform lengthy operation  to  customize
	      the  number  of  performed tests.  This function returns as an integer value of the -f option given to faltest or 1 if the -f option
	      was not given.

OPTIONS

       -c <cat>
	      Select this category and ignore the rest.

       -C <subcat>
	      Select this subcategory and ignore the rest.

       -d     Directory from where to load the unit tests. If not specified, it will be the current directory.

       -f <n> Set time factor to N. Some scripts may use this information to perform more loops or lengthy operations.

       -h     Show version and a short help.

       -l     List the selected tests and exit. Combine with -v to have the path of the tests, as -l only lists the script ID, its short name  and
	      the category/subcategory pair.

       -m     Do not compile in memory. Use disk temporary files.

       -M     Checks for memory leaks. This option makes faltest to report unclaimed memory after each script execution, and a final report at the
	      end of the tests. The check is extended to all the engine operations, so errors in the engine are detected too.

       -o <file>
	      Write final report to the given output file.

       -s     Perform module serialization test. Other than compiling the file, the module is also saved and then restored before being  executed.
	      This  allows one to check for errors in module serialization (that is, loading of .fam files). The operation is performed in memory,
	      unless the option -m is also specified.

       -S     Compile via assembly. This test the correct behavior of the assembler generator and compiler instead of the binary module generator.

       -t     Records and display timings. The statistics of compilation, linking and execution overall times are recorded and written as part	of
	      the report.

       -T     Records  timings() calls from scripts. This allows the scripts to declare their own performance ratings, and collects the results in
	      the final report.

       -v     Be verbose. Normally, execution and failures are sparsely reported. This is because the normal execution mode is meant for automated
	      runs. Tests can be executed by automated utilities and errors can be reported to system administrator by simple checks on the output
	      data.

	      A developer willing to fix a broken test must run that test alone with the -v enabled. A more complete error report (including  com-
	      pilation or execution errors, if they were the cause for the failure) will be then visualized. The -v options also allows one to see
	      the path of the original script, which is otherwise hidden (masked by the testsuite ID).

       -V     Prints version number and exits.

SAMPLE

       This is a simple and complete example from the Falcon benchmark suite.

       /*******************************************
       * Falcon direct benchmarks
       *
       * ID: 2a
       * Category: benchmark
       * Subcategory: calls
       * Short: Benchmark on function calls
       * Description:
       *    Performing repeated function calls and returns.
       *    This test calls a function without parameters.
       *
       * [/Description]
       ******************************************/

       loops = 1000000 * timeFactor()
       each = int(loops/10)

       function toBeCalled()
       end

       // getting time
       time = seconds()
       for i = 1 to loops

	  // perform the call
	  toBeCalled()

	  if i % each == 0
	     alive( i/loops*100 )
	  end
       end
       // taking end time
       time = seconds() - time

       // subtract alive time
       timeAlive = seconds()
       for i = 1 to loops
	  if i % each == 0
	     alive( i/loops*100 )
	  end
       end
       timeAlive = seconds() - timeAlive
       time -= timeAlive

       timings( time, loops )

       /* end of file */

FILES

       /usr/lib/libfalcon_engine.so
	      Default location of the Falcon Engine loadable module.

AUTHOR

       Giancarlo Niccolai <gc@falconpl.org>

SEE ALSO

       falcon(1)

LICENSE

       This document is released under the "GNU Free Documentation License, version 1.2".  On Debian systems, the complete text of the Free  Docu-
       mentation License, version 1.2, can be found in /usr/share/common-licenses/.

Falcon toolset							    April 2007								faltest(1)