[Igor Sobreira] Python coverage threshold

Ned Batchelder committed a new feature to coverage.py 2 days ago, it’s a way to verify coverage threshold. Basically a way to fail your tests if the coverage is not enough.
For now you need to install from the repository to get the new feature

$ pip install hg+https://bitbucket.org/ned/coveragepy#egg=coverage

This is how to use from the command line:

$ coverage run run_my_tests.py
... running all tests
$ coverage report --fail-under=100
... display the report
$ echo $?

If your coverage is not 100% your exit status will be 2. This will make your CI fail if your coverage is not enough :).
You can also verify coverage using the python API, in this case you need to verify the return value from report() function, here is an example:

cov = coverage.coverage(..)
ret = run_all_my_tests()
if ret == 0:
    covered = cov.report()
    assert covered > 100, "Not enough coverage"

I’ve created a decorator to make this easier:

def ensure_coverage(percentage, **cov_options):
    def decorator(function):
        def wrapper(*args, **kw):
            cov = coverage.coverage(branch=True, **cov_options)
            ret = function(*args, **kw)
            if ret == 0:
                covered = cov.report()
                assert covered >= percentage, \
                    "Not enough coverage: {0:.2f}%. You need at least {1}%".format(covered, percentage)
            return ret
        return wrapper
    return decorator

This is an usage example for this django app I’m working on:

@ensure_coverage(99, source=['filecabinet'], omit=['filecabinet/tests/*'])
def runtests():
    test_runner = get_runner(settings)()
    return test_runner.run_tests(['filecabinet'])

if __name__ == '__main__':

Here are the related commits, if you’re interested:


And the discussion on the issue: