[Igor Sobreira] Python coverage threshold

Published on: 05/11/2012 09:48h

Ned Batchelder committed a new feature to coverage.py 2 days ago, it’s a way to verify coverage threshold. Basically a way to fail your tests if the coverage is not enough.
For now you need to install from the repository to get the new feature

$ pip install hg+https://bitbucket.org/ned/coveragepy#egg=coverage

This is how to use from the command line:

$ coverage run run_my_tests.py ... running all tests$ coverage report --fail-under=100... display the report$ echo $?2

If your coverage is not 100% your exit status will be 2. This will make your CI fail if your coverage is not enough :)
You may also verify coverage using the python API, in this case you need to verify the return value from report() function, here is an example:

cov = coverage.coverage(..)cov.start()ret = run_all_my_tests()cov.stop()if ret == 0:    covered = cov.report()    assert covered > 100, "Not enough coverage"...

I’ve created a decorator to make this easier:

def ensure_coverage(percentage, **cov_options):    def decorator(function):        @wraps(function)        def wrapper(*args, **kw):            cov = coverage.coverage(branch=True, **cov_options)            cov.start()            ret = function(*args, **kw)            cov.stop()            if ret == 0:                covered = cov.report()                assert covered >= percentage, \                    "Not enough coverage: {0:.2f}%. You need at least {1}%".format(covered, percentage)            return ret        return wrapper    return decorator

This is an usage example for this django app I’m working on:

@ensure_coverage(99, source=['filecabinet'], omit=['filecabinet/tests/*'])def runtests():    test_runner = get_runner(settings)()    return test_runner.run_tests(['filecabinet'])

if __name__ == '__main__':    sys.exit(runtests())

Here are the related commits, if you’re interested:

https://bitbucket.org/ned/coveragepy/changeset/7ea709fc4c1190cf0ffe0aba1a49e6fffe683d2f
https://bitbucket.org/ned/coveragepy/changeset/90014f4defd336f05851bdfc01c2b5af60a933c9
https://bitbucket.org/ned/coveragepy/changeset/ba7267fe525001dba99ed1c2c9d11f0724ad9950

And the discussion on the issue:

https://bitbucket.org/ned/coveragepy/issue/139/easy-check-for-a-certain-coverage-in-tests

By Igor Sobreira