Use time elapsed as assertion in unit tests

Question

I am facing a situation in which I want to ensure that if a method is modified it does not takes more than X ms to run (essentially if that function is slower, it will slow down the search results on our web page, and that has a bad impact on sales).

We have unit tests (in particular the code is done under python and we use py.test), the first idea is to assert that if the function did not execute in less than X ms, then mark the test as failed (or raise a warning).

However, this feels dangerous (not all computers are the same speed, for example), and also I am not quite sure that's the job of a unit test.

Has someone faced a similar situation? For me speed is a feature, and I want to ensure that such feature is not lost in the future as the code evolves.

If unit testing is not the answer, which other alternatives would you recommend?

Thanks

Answer 1

You could look into using the pytest-timeout plugin for a start. It currently only allows specifying seconds (as floats). But it could be extended to work with more adaptable timings such as pystone numbers so there would be a measurement of how fast your computer performs the pystone benchmark which could be cached and then you'd specify how many "pystones" your test(s) is allowed to take.

Answer 2

If you have a dedicated build/test server, or roughly equivalent hardware between devs, it's fine to have a time limit in ms. Repeatability is key - you may be able to write something allowing devs to do a "baseline" test run on their machines that subsequent results are then compared to.

There's pyunitperf module available that features time limits as well as load testing. It hasn't been updated for a long time (2009) but would be a good place to start if you're going to roll your own.

And remember that a test time in ms itself isn't going to tell you anything - but as long as you're only using this to compare performance across revisions, it should be okay.