Fork me on GitHub

test-framework by Max Bolingbroke

A parallel test framework for combining tests made using QuickCheck and HUnit and much more...

Description

A common testing framework that allows tests such as QuickCheck properties and HUnit test cases to used together easily. Furthermore, it enhances both styles of test with all of these great features:

If you like test-framework you may also want to check out Oscar Finnsson's test-framework-th package for generating your test groups programatically.

Example

The latest version of the example is always available online:

-- Adapted from an example by Don Stewart. For licensing information
-- please see the file "example/Test/Framework/Example.lhs" in the source tree.
import Test.Framework (defaultMain, testGroup)
import Test.Framework.Providers.HUnit
import Test.Framework.Providers.QuickCheck (testProperty)

import Test.QuickCheck
import Test.HUnit

import Data.List


main = defaultMain tests

tests = [
        testGroup "Sorting Group 1" [
                testProperty "sort1" prop_sort1,
                testProperty "sort2" prop_sort2,
                testProperty "sort3" prop_sort3
            ],
        testGroup "Sorting Group 2" [
                testProperty "sort4" prop_sort4,
                testProperty "sort5" prop_sort5,
                testProperty "sort6" prop_sort6,
                testCase "sort7" test_sort7,
                testCase "sort8" test_sort8
            ]
    ]


prop_sort1 xs = sort xs == sortBy compare xs
  where types = (xs :: [Int])

prop_sort2 xs =
        (not (null xs)) ==>
        (head (sort xs) == minimum xs)
  where types = (xs :: [Int])

prop_sort3 xs = (not (null xs)) ==>
        last (sort xs) == maximum xs
  where types = (xs :: [Int])

prop_sort4 xs ys =
        (not (null xs)) ==>
        (not (null ys)) ==>
        (head (sort (xs ++ ys)) == min (minimum xs) (minimum ys))
  where types = (xs :: [Int], ys :: [Int])

prop_sort5 xs ys =
        (not (null xs)) ==>
        (not (null ys)) ==>
        (head (sort (xs ++ ys)) == max (maximum xs) (maximum ys))
  where types = (xs :: [Int], ys :: [Int])

prop_sort6 xs ys =
        (not (null xs)) ==>
        (not (null ys)) ==>
        (last (sort (xs ++ ys)) == max (maximum xs) (maximum ys))
  where types = (xs :: [Int], ys :: [Int])

test_sort7 = sort [8, 7, 2, 5, 4, 9, 6, 1, 0, 3] @?= [0..9]

test_sort8 = error "This test deliberately contains a user error"

We can run these tests from the command line (in parallel!) like so:

$ ghc -package test-framework -package test-framework-quickcheck \
    -package test-framework-hunit -threaded Example.hs -o Example
$ ./Example --maximum-generated-tests=5000 +RTS -N2

FAQ

I tried to run the example, and got an error about not having a Testable instance

You may get an error like the following:

No instance for (QuickCheck-1.2.0.0:Test.QuickCheck.Testable
                      (Gen Prop))
     arising from a use of `testProperty' at Example.lhs:68:16-46

The reason is that GHC is building the example (which uses QuickCheck 1) with the QuickCheck 2 library. You can fix this by running:

$ ghc-pkg hide QuickCheck-2.1.0.1

You may need adjust the version number to match the one on your machine, as reported by ghc-pkg list. Alternatively, you can supply the -package QuickCheck-1.2.0.0 flag when invoking GHC (again you may need to change the version number to match your installed QuickCheck version).

I have some tests that just call error but the tests pass

This happens because GHC finds it convenient to eta-expand the calls to error, so we can no longer observe the failure. This is dodgy behaviour by GHC, but it's unlikely to change any time soon, so the workaround is to compile your tests with -O0 (the default).

If you really do want to compile your tests with optimisations enabled, you should also be able to supply the -fno-full-laziness flag along with your -O1 or -O2 flag. However, if new optimisations are added to GHC they may break this workaround - it's probably safest to turn optimisation off altogether.

Providers

The framework is built with an extensible architecture that allows anyone to add their own test "provider". All of the supplied providers are built using this mechanism, and so you probably want to install not only the infrastructure package (test-framework), but also the commonly-used providers test-framework-quickcheck and test-framework-hunit.

The Console Test Runner

A description of the options available can be obtained by using the option --help on the command line.

The test-selection syntax for use with the -s command line option is based on that of shell globs or Git .gitignore files. Test patterns consist of the following:

For example, group/*1 matches group/test1 but not group/subgroup/test1, whereas both examples would be matched by group/**1. A leading slash matches the beginning of the test path; for example, /test* matches test1 but not group/test1.

A test will be run if it matches any of the patterns supplied with -t.

Documentation

Haddock documentation is available at Hackage: