David R. MacIver's Blog: I just want to brag a bit about the Hypothesis test suite

I just want to brag a bit about the Hypothesis test suite

12 January 2015

There’s not much content to this post other than “look at how awesome my test suite is”. But it’s pretty awesome. It:

Has 100% branch coverage
Fails the build if it ever doesn’t have 100% branch coverage
Has about a 3:2 ratio of lines of test code to library code
Also runs the tests against a version installed from a package. This both tests the installer and also tests without coverage in case coverage changes the semantics (which it probably won’t now but definitely will in future)
Runs across the three major python versions the library supports (2.7, 3.3, 3.4)
Has statistical tests about the properties of the distributions it produces
Runs a linter over the code to make sure that it’s pyflakes and pep8 compliant
Runs a linter over the README to make sure that it parses correctly as RST
Has a self test where it uses hypothesis to test hypothesis. This found real bugs in the library, despite the aforementioned coverage and code ratio.

This is probably the best tested code I’ve ever written, and I’m still morally certain it has at least one major bug in it.

I don’t know what that bug is of course, or I’d have fixed it, but every single time I’ve gone “Wow this code is really well tested. I bet it’s super solid” it has been followed by “Hmm. I could do this thing to prove it’s super solid. Oh look there’s a bug” .

OK, so this started out as me bragging about how great my test suite was but now I’m super depressed about the state of software all over again. I think of myself as being atypically good at producing correct code - not in some mythical 10x sense where I’m vastly better than everyone else, but I feel pretty comfortable in saying I’m better than average. This is probably the most well tested code I’ve ever written, and it’s still buggy. Where does that leave the majority of code found in the wild?

Comments

The pain of randomized testing | David R. MacIver on 2015-01-13 11:22:05:

[…] my best all the time to try to make Hypothesis’s test suite less flaky while still keeping it as thorough as it currently is (ideally more thorough given that every time I figure out a new method for testing hypothesis I […]

The things you forget to test | David R. MacIver on 2015-01-15 09:08:20:

[…] mentioned recently, Hypothesis has 100% branch […]