al2-test-runner -- Alternative way of running rackunit test suites

8.12

al2-test-runner – Alternative way of running rackunit test suites🔗ℹ

Alex Harsányi

This package provides an alternative way of running rackunit tests, providing better visibility and improved reporting of test results. The run-tests function is a drop-in replacement of the equivalent function from rackunit/textui, providing the following benefits:

Results from test runs are reported to the standard output, even if they are successful. This allows better visibility of what tests are actually run, along with their execution time
Test results are written to an output file, using JUnit compatible format. This allows test results to be imported into various management tools for analyzing trends such as test durations and failure rates across different builds
Tests can be explicitly skipped, and such tests are reported as skipped – this allows better visibility of what tests are not run, when compared to just commenting them out in the source code.

The package does not define another testing framework, instead it runs rackunit tests which are organized in test-suites and test-cases.

1 Motivating Example🔗ℹ

When working on an application or package, you might disable a broken test temporarily, but once you do, there is no longer an indication that the test has been disabled. The test suite passes, and if there is a large number of tests, it is easy to miss that some were disabled:

> (require rackunit rackunit/text-ui)
> (define a-test-suite
    (test-suite "A Test Suite"
      (test-case "First Test Case" (check-equal? 1 1))
      ; (test-case "Second Test Case" (check-equal? 1 0))
      (test-case "Third Test Case" (check-equal? 1 1))))
> (run-tests a-test-suite 'verbose)
2 success(es) 0 failure(s) 0 error(s) 2 test(s) run
0

The commented out test might be easy to spot in the previous example, but in a bigger application with many tests, it is easy to miss. The al2-test-runner package provides an alternative: the test can be disabled by adding it to the exclusion list in run-tests. The test is still known to the test system and it is now reported as skipped.

> (require rackunit al2-test-runner)
> (define a-test-suite
    (test-suite "A Test Suite"
      (test-case "First Test Case" (check-equal? 1 1))
      (test-case "Second Test Case" (check-equal? 1 0))
      (test-case "Third Test Case" (check-equal? 1 1))))
> (run-tests
    #:package "my-package"
    #:results-file "my-package-test-results.xml"
    ; Second test case is broken, will fix it later
    #:exclude '(("A Test Suite" "Second Test Case"))
    a-test-suite)
*** Testsuite A Test Suite
First Test Case: ok (0.03 ms)
Second Test Case: skipped
Third Test Case: ok (0.02 ms)
*** Testsuite A Test Suite completed in 0.05 ms
*** Total tests: 3 ; failures: 0 ; errors: 0 ; skipped: 1
*** Writing results to my-package-test-results.xml

In addition, this version of run-tests will print out all the tests that are run along with their pass/fail status and will also write this information to a file, in standard JUnit test format. This file can be imported in various test management systems.

2 API Documentation🔗ℹ

procedure
(run-tests [ #:package package
#:results-file results-file
#:only only
#:exclude exclude]
test-suite ...) → any/c
  package : string? = "unnamed package"
  results-file : (or/c path-string? #f) = #f
  only : (or/c #f (listof (listof string?))) = #f
  exclude : (or/c #f (listof (listof string?))) = #f
  test-suite : test-suite?

Run the tests in test-suites and report the results to the standard output. For each test case, the name followed by the strings "ok", "failed", "error" or "skipped" is printed, along with the time in milliseconds it took to execute the test.

package is a string that is used as a package name when results are written to file. JUnit tests are grouped into packages, and the XML file format expects that, but there is no higher construct than a test-suite in rackunit, so the package name needs to be provided as a parameter to run-tests.

When results-file is specified, the test results are written to the specified file in JUnit XML format. If a file name is not specified, the results are not written to file.

When only is present, it needs to be a list of test suite names followed by test case names, and only these tests will be run. This is intended to be used when debugging a single test in a larger test suite. Tests that are not run will be reported as skipped. If the list of test cases is empty, and. only the test suite name is specified, all the tests cases in that test suite are run.

Here is an example of how to run only one test case in the test suite:

> (require rackunit al2-test-runner)
> (define a-test-suite
    (test-suite "A Test Suite"
      (test-case "First Test Case" (check-equal? 1 1))
      (test-case "Second Test Case" (check-equal? 1 1))
      (test-case "Third Test Case" (check-equal? 1 1))))
> (run-tests
    #:package "my-package"
    #:results-file "my-package-test-results.xml"
    #:only '(("A Test Suite" "First Test Case"))
    a-test-suite)
*** Testsuite A Test Suite
First Test Case: ok (0.03 ms)
Second Test Case: skipped
Third Test Case: skipped
*** Testsuite A Test Suite completed in 0.03 ms
*** Total tests: 3 ; failures: 0 ; errors: 0 ; skipped: 2
*** Writing results to my-package-test-results.xml

When exclude is preset, it needs to be a list of test case names followed by test suite names. These tests will not be executed, and instead will be reported as skipped both on the standard output and in the output file. Note that tests can also be skipped using skip-test. This is indented for disabling tests which do not pass and they cannot be fixed immediately. If the list of test cases is empty, and. only the test suite name is specified, all the tests cases in that test suite are excluded.

Here is an example, that will not run "A Test Case":

> (require rackunit al2-test-runner)
> (define a-test-suite
    (test-suite "A Test Suite"
      (test-case "First Test Case" (check-equal? 1 1))
      (test-case "Second Test Case" (check-equal? 1 1))
      (test-case "Third Test Case" (check-equal? 1 1))))
> (run-tests
    #:package "my-package"
    #:results-file "my-package-test-results.xml"
    #:exclude '(("A Test Suite" "First Test Case"))
    a-test-suite)
*** Testsuite A Test Suite
First Test Case: skipped
Second Test Case: ok (0.06 ms)
Third Test Case: ok (0.03 ms)
*** Testsuite A Test Suite completed in 0.08 ms
*** Total tests: 3 ; failures: 0 ; errors: 0 ; skipped: 1
*** Writing results to my-package-test-results.xml

procedure
(skip-test) → any/c

This function can be used inside test cases to skip a test because some required testing condition cannot be met. It can be used, for example, when tests require test data files and these data files are not available and it is a better indicator that a test suite did not actually fail but it didn’t pass either.

Here is an example where the second test case is skipped:

> (require rackunit al2-test-runner)
> (define a-test-suite
    (test-suite "A Test Suite"
      (test-case "First Test Case" (check-equal? 1 1))
      (test-case "Second Test Case" (unless (= 1 0) (skip-test)))
      (test-case "Third Test Case" (check-equal? 1 1))))
> (run-tests
    #:package "my-package"
    #:results-file "my-package-test-results.xml"
    a-test-suite)
*** Testsuite A Test Suite
First Test Case: ok (0.05 ms)
Second Test Case: skipped (0.03 ms)
Third Test Case: ok (0.03 ms)
*** Testsuite A Test Suite completed in 0.11 ms
*** Total tests: 3 ; failures: 0 ; errors: 0 ; skipped: 1
*** Writing results to my-package-test-results.xml