rackcheck: Property-based Testing

8.12

rackcheck: Property-based Testing🔗ℹ

Rackcheck is a property-based testing library for Racket with support for shrinking.

1 What about quickcheck?🔗ℹ

I initially started by forking the quickcheck library to add support for shrinking, but found that I would have to make many breaking changes to get shrinking to work the way I wanted so I decided to start from scratch instead.

2 Reference🔗ℹ

(require rackcheck)

package: rackcheck-lib

2.1 Generators🔗ℹ

Generators produce arbitrary values based upon certain constraints. The generators provided by this library can be mixed and matched in order to produce complex values suitable for any domain.

By convention, all generators and generator combinators are prefixed with gen:.

2.1.1 Debugging🔗ℹ

The following functions come in handy when debugging generators. Don’t use them to produce values for your tests.

procedure
(sample g n [rng]) → (listof any/c)
  g : gen?
  n : exact-positive-integer?
   rng : pseudo-random-generator?
= (current-pseudo-random-generator)

Samples n values from g.

procedure
(shrink g
size
[ rng
#:limit limit
#:max-depth max-depth]) → (listof any/c)
  g : gen?
  size : exact-nonnegative-integer?
   rng : pseudo-random-generator?
= (current-pseudo-random-generator)
  limit : (or/c #f exact-nonnegative-integer?) = #f
  max-depth : (or/c #f exact-nonnegative-integer?) = #f

Produces a value from g and evaluates its shrink tree. The evaluation searches for up to limit shrinks of each term, up to a depth of max-depth, or unbounded if #f.

2.1.2 Core Combinators🔗ℹ

value
generator/c
: (-> pseudo-random-generator? exact-nonnegative-integer? shrink-tree?)

The contract for generator functions. Generator functions produce a lazily evaluated tree containing the different ways that the generated value can be shrunk.

In general, you won’t have to write generator functions yourself. Instead, you’ll use the generators and combinators provided by this library to generate values for your domain. That said, there may be cases where you want to tightly control how values are generated or shrunk and that’s when you might reach for a custom generator.

procedure
(gen? v) → boolean?
v : any/c
procedure
(make-gen f) → gen?
f : generator/c

gen? returns #t when v is a generator value and make-gen creates a new generator from a generator function.

Use custom generators sparingly. They’re one area where we might break backward compatibility if we come up with a better implementation.

Changed in version 2.0 of package rackcheck-lib: Generators now use shrink trees for shrinking.

procedure
(gen:const v) → gen?
v : any/c

Creates a generator that always returns v.

procedure
(gen:map g f) → gen?
g : gen?
f : (-> any/c any/c)

Creates a generator that transforms the values generated by g by applying them to f before returning them.

procedure
(gen:bind g f) → gen?
g : gen?
f : (-> any/c gen?)

Creates a generator that depends on the values produced by g.

When shrinking, the shrinking of the output of g is prioritized over the shrinking of the output of f.

> (define gen:list-of-trues
    (gen:bind
     gen:natural
     (lambda (len)
       (apply gen:tuple (make-list len (gen:const #t))))))
> (sample gen:list-of-trues 5)
'(() (#t) (#t #t #t #t) (#t #t #t #t #t #t #t #t #t) (#t #t))
> (shrink gen:list-of-trues 5)
'((#t #t) (()) ((#t) ...))

procedure
(gen:filter g p [max-attempts]) → gen?
  g : gen?
  p : (-> any/c boolean?)
  max-attempts : (or/c exact-positive-integer? +inf.0) = 1000

Produces a generator that repeatedly generates values using g until the result of p applied to one of those values is #t or the number of attempts exceeds max-attempts.

An exception is raised when the generator runs out of attempts.

This is a very brute-force way of generating values and you should avoid using it as much as possible, especially if the range of outputs is very small compared to the domain. Take a generator for non-empty strings as an example. Instead of:

> (gen:filter
   (gen:string gen:char-alphanumeric)
   (lambda (s)
     (not (string=? s ""))))
#<procedure:...ck-lib/gen/core.rkt:114:3>

Write:

> (gen:let ([hd gen:char-alphanumeric]
[tl (gen:string gen:char-alphanumeric)])
(string-append (string hd) tl))
#<procedure:...ck-lib/gen/core.rkt:100:3>

The latter takes a little more effort to write, but it doesn’t depend on the whims of the random number generator and will always generate a non-empty string on the first try.

procedure
(gen:choice g ...+) → gen?
g : gen?

Produces a generator that generates values by randomly choosing one of the passed-in generators.

procedure
(gen:sized f) → gen?
f : (-> exact-nonnegative-integer? gen?)

Creates a generator that uses the generator’s size argument.

procedure
(gen:resize g size) → gen?
g : gen?
size : exact-nonnegative-integer?

Creates a generator that overrides the generator’s size with size.

procedure
(gen:scale g f) → gen?
  g : gen?
   f :
(-> exact-nonnegative-integer?
    exact-nonnegative-integer?)

Creates a generator that modifies the size by applying f.

procedure
(gen:no-shrink g) → gen?
g : gen?

Creates a generator based on g that never shrinks.

procedure
(gen:with-shrink g proc) → gen?
g : gen?
proc : shrink-proc/c

Creates a generator that overrides the default shrinking with recursive applications of proc to the generator output.

syntax
(gen:let ([id gen-expr] ...+) body ...+)

Provides a convenient syntax for creating generators that depend on one or more other generators.

> (define gen:list-of-trues-2
(gen:let ([len gen:natural])
(make-list len #t)))
> (sample gen:list-of-trues-2 5)
'(() (#t) (#t #t #t #t) (#t #t #t #t #t #t #t #t #t) (#t #t))
> (shrink gen:list-of-trues-2 5 #:max-depth #f)
'((#t #t) (()) ((#t) (())))

syntax
(gen:delay gen-expr)

Creates a generator that wraps and delays the execution of gen-expr until it is called. This is handy for when you need to write recursive generators.

2.1.3 Basic Generators🔗ℹ

value
gen:natural : gen?

Generates natural numbers.

> (sample gen:natural)
'(0 0 4 5 8 20 22 39 43 20)
> (shrink gen:natural 5)
'(3 (0) (2 ...))

procedure
(gen:integer-in lo hi) → gen?
lo : exact-integer?
hi : exact-integer?

Creates a generator that produces exact integers between lo and hi, inclusive. A contract error is raised if lo is greater than hi.

> (sample (gen:integer-in 1 255))
'(75 46 239 152 124 200 155 202 172 65)
> (shrink (gen:integer-in 1 255) 5)
'(159 (80 ...) (120 ...) (140 ...) (150 ...) (155 ...) (157 ...) (158 ...))
> (shrink (gen:integer-in -99 0) 5)
'(-45 (0) (-23 ...) (-34 ...) (-40 ...) (-43 ...) (-44 ...))

Changed in version 2.1 of package rackcheck-lib: The delta between hi and lo may now exceed 4294967087.

value
gen:real : gen?

Generates real numbers between 0 and 1, inclusive. Real numbers do not currently shrink, but this may change in the future.

> (sample gen:real)
'(0.2904158091187683
  0.17902984405826025
  0.9348212358175817
  0.592848361775386
  0.4846099332903666
  0.7816821100632378
  0.6078124617750272
  0.788902313469835
  0.6710271948421507
  0.25158978983077135)
> (shrink gen:real 5)
'(0.6227904093778704)

procedure
(gen:one-of choices [equal?-proc]) → gen?
choices : (non-empty-listof any/c)
equal?-proc : (-> any/c any/c boolean?) = equal?

Creates a generator that produces values randomly selected from choices. When shrinking, removes previously-chosen values from the set using equal?-proc.

> (define gen:letters
(gen:one-of '(a b c)))
> (sample gen:letters)
'(c a b a c a c c b b)
> (shrink gen:letters 5 #:max-depth #f)
'(a (c (b)))

value
gen:boolean : gen?

Generates boolean values.

> (sample gen:boolean)
'(#f #f #t #t #f #t #t #t #t #f)
> (shrink gen:boolean 10)
'(#t (#f))

value
gen:char : gen?

Generates ASCII characters.

> (sample gen:char)
'(#\J #\- #\ï #\u0097 #\| #\È #\u009B #\É #\« #\@)
> (shrink gen:char 5)
'(#\u009F
  (#\nul)
  (#\P ...)
  (#\x ...)
  (#\u008C ...)
  (#\u0096 ...)
  (#\u009B ...)
  (#\u009D ...)
  (#\u009E ...))

value
gen:char-letter : gen?

Generates ASCII letters.

> (sample gen:char-letter)
'(#\e #\P #\u #\U #\G #\O #\g #\E #\u #\o)
> (shrink gen:char-letter 5)
'(#\X (#\B ...) (#\M ...) (#\S ...) (#\V ...) (#\W ...))

value
gen:char-digit : gen?

Generates ASCII digits.

> (sample gen:char-digit)
'(#\2 #\1 #\9 #\5 #\4 #\7 #\6 #\7 #\6 #\2)
> (shrink gen:char-digit 5)
'(#\6 (#\0) (#\3 ...) (#\5 ...))

value
gen:char-alphanumeric : gen?

Generates alphanumeric ASCII characters.

> (sample gen:char-alphanumeric)
'(#\1 #\M #\U #\q #\g #\d #\o #\H #\2 #\7)
> (shrink gen:char-alphanumeric 5)
'(#\1 (#\0))

procedure
(gen:tuple g ...) → gen?
g : gen?

Creates a generator that produces heterogeneous lists where the elements are created by generating values from each g in sequence.

> (sample (gen:tuple gen:natural gen:boolean))
'((0 #f) (1 #t) (2 #t) (6 #t) (11 #f) (16 #t) (9 #f) (46 #f) (8 #t) (38 #t))
> (shrink (gen:tuple gen:natural gen:boolean) 5)
'((3 #t) ((0 #t) ...) ((2 #t) ...) ((3 #f) ...))

procedure
(gen:list g [#:max-length max-len]) → gen?
g : gen?
max-len : exact-nonnegative-integer? = 128

Creates a generator that produces lists of random lengths where every element is generated using g. Shrinks by removing elements from the list, then by shrinking individual elements of the list.

> (sample (gen:list gen:natural) 5)
'(() () (2 2 3 3) (6 2 6 5 2 2 9) (2 13))
> (shrink (gen:list gen:natural) 5)
'((3 3)
  (())
  ((3) ...)
  ((3) ...)
  ((0 3) ...)
  ((2 3) ...)
  ((3 0) ...)
  ((3 2) ...))

procedure
(gen:vector g [#:max-length max-len]) → gen?
g : gen?
max-len : exact-nonnegative-integer? = 128

Like gen:list but for vector?s.

> (sample (gen:vector gen:natural) 5)
'(#() #() #(2 2 3 3) #(6 2 6 5 2 2 9) #(2 13))
> (shrink (gen:vector gen:natural) 5)
'(#(3 3)
  (#())
  (#(3) ...)
  (#(3) ...)
  (#(0 3) ...)
  (#(2 3) ...)
  (#(3 0) ...)
  (#(3 2) ...))

procedure
(gen:bytes [g #:max-length max-len]) → gen?
g : gen? = (gen:integer-in 0 255)
max-len : exact-nonnegative-integer? = 128

Like gen:list but for bytes?s. Raises a contract error if g produces anything other than integers in the range 0 to 255 inclusive.

> (sample (gen:bytes) 5)
'(#"" #"" #"\227|\310\233" #"\253@\237\214>A\354" #"#\312")
> (shrink (gen:bytes) 5)
'(#"\222\246"
  (#"")
  (#"\246" ...)
  (#"\222" ...)
  (#"\0\246" ...)
  (#"I\246" ...)
  (#"n\246" ...)
  (#"\200\246" ...)
  (#"\211\246" ...)
  (#"\216\246" ...)
  (#"\220\246" ...)
  (#"\221\246" ...)
  (#"\222\0" ...)
  (#"\222S" ...)
  (#"\222}" ...)
  (#"\222\222" ...)
  (#"\222\234" ...)
  (#"\222\241" ...)
  (#"\222\244" ...)
  (#"\222\245" ...))

procedure
(gen:string [g #:max-length max-len]) → gen?
g : gen? = gen:char
max-len : exact-nonnegative-integer? = 128

Like gen:list but for string?s. Raises a contract error if g produces anything other than char? values.

> (sample (gen:string gen:char-letter) 5)
'("" "" "MPRq" "gEuoX" "fuee")
> (shrink (gen:string gen:char-letter) 5)
'("")

procedure
(gen:symbol [g #:max-length max-len]) → gen?
g : gen? = gen:char
max-len : exact-nonnegative-integer? = 128

Like gen:string but for symbol?s. Raises a contract error if g produces anything other than char? values.

> (sample (gen:symbol gen:char-letter) 5)
'(|| || MPRq gEuoX fuee)
> (shrink (gen:symbol gen:char-letter) 5)
'(||)

procedure
(gen:hash k g ...+ ...+) → gen?
  k : any/c
  g : gen?
procedure
(gen:hasheq k g ...+ ...+) → gen?
  k : any/c
  g : gen?
procedure
(gen:hasheqv k g ...+ ...+) → gen?
  k : any/c
  g : gen?

These functions create generators that produce hashes, hasheqs and hasheqvs, respectively, where each key maps to a value generated from its associated generator.

> (sample (gen:hasheq 'a gen:natural 'b (gen:string gen:char-letter)) 5)
'(#hasheq((a . 0) (b . ""))
  #hasheq((a . 1) (b . "M"))
  #hasheq((a . 1) (b . "RqG"))
  #hasheq((a . 1) (b . "dMQHfueeb"))
  #hasheq((a . 7) (b . "asTNCCYEwZCkc")))
> (shrink (gen:hasheq 'a gen:natural 'b (gen:string gen:char-letter)) 5)
'(#hasheq((a . 4) (b . ""))
  (#hasheq((a . 0) (b . "")))
  (#hasheq((a . 2) (b . "")) ...)
  (#hasheq((a . 3) (b . "")) ...))

procedure
(gen:frequency frequencies) → gen?
frequencies : (non-empty-listof (cons/c exact-nonnegative-integer? gen?))

Creates a generator that generates values using a generator that is randomly picked from frequencies. Generators with a higher weight will get picked more often. The sum of the cars of the frequencies must be greater than zero.

> (sample (gen:frequency `((5 . ,gen:natural)
                           (2 . ,gen:boolean))))
'(0 #f #t #f 7 18 35 #t 19 10)
> (shrink (gen:frequency `((5 . ,gen:natural)
                           (2 . ,gen:boolean)))
          5)
'(1 (0))

2.1.4 Unicode Generators🔗ℹ

(require rackcheck/gen/unicode)

package: rackcheck-lib

value
gen:unicode : gen?
value
gen:unicode-letter : gen?
value
gen:unicode-mark : gen?
value
gen:unicode-number : gen?
value
gen:unicode-punctuation : gen?
value
gen:unicode-symbol : gen?
value
gen:unicode-separator : gen?

These generators produce valid unicode char?s.

> (sample gen:unicode)
'(#\U0003C314
  #\耎
  #\U000D7AFB
  #\ꩧ
  #\㙗
  #\登
  #\U00050685
  #\⏲
  #\U000DA289
  #\U000A1C2C)
> (sample gen:unicode-letter)
'(#\𘮙 #\屔 #\鱕 #\𣜋 #\𣄬 #\𡐦 #\𫆕 #\𐽱 #\潇 #\𠔑)
> (sample gen:unicode-mark)
'(#\𖿱 #\ࣺ #\֫ #\𑶍 #\𞓬 #\𑴶 #\𞣐 #\ॅ #\󠅢 #\𑄀)
> (sample gen:unicode-number)
'(#\𐩄 #\𐧜 #\𑁡 #\᮶ #\𐹺 #\Ⅻ #\𑁭 #\𒑨 #\൸ #\𑣥)
> (sample gen:unicode-punctuation)
'(#\· #\᥄ #\᳂ #\𑩃 #\― #\॥ #\｢ #\‿ #\𝪉 #\܋)
> (sample gen:unicode-symbol)
'(#\㏨ #\🪬 #\⪅ #\𜾶 #\¦ #\𝢏 #\⬣ #\⭍ #\⮒ #\◴)
> (sample gen:unicode-separator)
'(#\space
  #\u2001
  #\u2002
  #\u2028
  #\u2008
  #\u3000
  #\u2001
  #\u3000
  #\u1680
  #\u2004)

2.2 Properties🔗ℹ

procedure
(property? v) → boolean?
v : any/c

Returns #t when v is a property.

syntax
(property maybe-name
([id gen-expr] ...)
body ...+)

maybe-name =
| name-id
| #:name name-expr

Declares a property where the inputs are one or more generators.

> (property ([xs (gen:list gen:natural)])
(check-equal? (reverse (reverse xs)) xs))
#<prop>

syntax
(define-property name
([id gen-expr] ...)
body ...+)

A shorthand for (define name (property name ([id gen-expr] ...) body ...+)).

syntax
(check-property maybe-config prop-expr)

maybe-config =
| config-expr

Tries to falsify the property p according to the config. If not provided, then a default configuration with a random seed value is used.

> (check-property
   (property ([xs (gen:list gen:natural)])
     (check-equal? (reverse (reverse xs)) xs)))
  ✓ property unnamed passed 100 tests.
> (check-property
   (property ([xs (gen:list gen:natural)])
     (check-equal? (reverse xs) xs)))
--------------------
FAILURE
location:   eval:55:0
name:       unnamed
seed:       2040279040
actual:     '(1 0)
expected:   '(0 1)
Failed after 4 tests:
  xs = (9 3 7 3 8)
Shrunk:
  xs = (0 1)
--------------------

procedure
(property-name property) → any/c
property : any/c

Returns name of the property.

> (define-property prop-list-map-identity
([xs (gen:list gen:natural)])
(check-equal? (map values xs) xs))
> (property-name prop-list-map-identity)
'prop-list-map-identity

procedure
(label! s) → void?
s : (or/c false/c string?)

Keeps track of how many times s appears in the current set of tests. Use this to classify and keep track of what categories the inputs to your properties fall under.

Does nothing when s is #f.

> (check-property
   (property
    ([a gen:natural]
     [b gen:natural])
    (label!
     (case a
      [(0)  "zero"]
      [else "non-zero"]))
    (+ a b)))
  ✓ property unnamed passed 100 tests.
  Labels:
  ├ 99.00% non-zero
  └  1.00% zero

procedure
(config? v) → boolean?
v : any/c

Returns #t when v is a config value.

procedure
(make-config [ #:seed seed
#:tests tests
#:size size
#:deadline deadline]) → config?
  seed : (integer-in 0 (sub1 (expt 2 31))) = ...
  tests : exact-positive-integer? = 100
   size : (-> exact-positive-integer? exact-nonnegative-integer?)
= (lambda (n) (expt (sub1 n) 2))
   deadline : (>=/c 0)
= (+ (current-inexact-milliseconds) (* 60 1000))

Creates values that control the behavior of check-property.

2.3 Shrink Trees🔗ℹ

(require rackcheck/shrink-tree)

package: rackcheck-lib

Added in version 2.0 of package rackcheck-lib.

A shrink tree is a tree containing a value and the ways it can be shrunk. Shrink trees are lazy.

value
shrink-proc/c : (-> any/c stream?)

The contract for shrinking procedures. A shrinking procedure receives a value from the shrink tree and must produce a stream of subsequent shrinks of that value.

procedure
(shrink-tree? v) → boolean?
v : any/c

Returns #t when v is a shrink tree.

procedure
(make-shrink-tree init-v [proc]) → shrink-tree?
init-v : any/c
proc : shrink-proc/c = (λ (_) empty-stream)

Creates a new shrink-tree from init-v, and a procedure for shrinking it. The proc argument may be omitted, in which case the shrink tree will not shrink past the initial value.

procedure
(shrink-tree-map tree proc) → shrink-tree?
tree : shrink-tree?
proc : (-> any/c any/c)

Returns a new shrink tree whose values will be mapped via proc when produced.