CODEWITHSUNDEEP
pythonpytestpython-hypothesis
tahsmith
tahsmith

Reputation: 1723

Why does my simple, finite hypothesis test never stop?

I am running a test suite with hypothesis-4.24.6 and pytest-5.0.0. My test has a finite set of possible inputs, but hypothesis never finishes testing.

I have reduced it to the following minimal example, which I run as pytest test.py

from hypothesis import given
import hypothesis.strategies as st


@given(x=st.just(0)
         | st.just(1),
       y=st.just(0)
         | st.just(1)
         | st.just(2))
def test_x_y(x, y):
    assert True

I would expect it to try all six combinations here and then succeed. Or possibly a small multiple of that to check for flakiness. Instead it runs indefinitely, (after about 15 mins of testing I kill it.)

If I interrupt the test, back traces seem to show it just continuously generating new examples.

What have I done wrong here?

Upvotes: 5

Views: 977

Answers (2)

Zac Hatfield-Dodds
Zac Hatfield-Dodds

Reputation: 3003

This bug was fixed in Hypothesis 4.26.2, or at least we thought so; it's actually fixed in 4.26.3 though: https://hypothesis.readthedocs.io/en/latest/changes.html#v4-26-3

Upvotes: 1

301_Moved_Permanently
301_Moved_Permanently

Reputation: 4186

This seems to be connected to the amount of successful tests hypothesis tries to generate:

>>> from hypothesis import given, strategies as st
>>> @given(st.integers(0,1), st.integers(0,2))
... def test(x, y):
...   print(x, y)
...   assert True
... 
>>> test()
0 0
1 1
1 0
1 2
1 1
0 1
0 0
1 2
0 2
0 2
1 0
1 2
0 1
0 1
1 2
[snip…]

See, this part of the docs, for instance, the default amount of successful test cases should be 100. So trying to generate more and more data to only restrict to 6 cases is rapidly failing to find one of these 6 cases.

The simplest approach can be to just limit the amount of examples needed for this test to pass:

>>> from hypothesis import settings
>>> @settings(max_examples=30)
... @given(st.integers(0,1), st.integers(0,2))
... def test(x, y):
...   print(x, y)
...   assert True
... 
>>> test()
0 0
1 1
1 0
0 2
1 2
0 1
0 1
1 1
1 0
1 1
0 1
1 2
1 1
0 0
0 2
0 2
0 0
1 2
1 0
0 1
1 0
1 0
0 1
1 2
1 1
0 2
0 0
1 2
0 0
0 2

An other approach, given the few amount of test cases, would be to explicit them all using @example and ask hypothesis to only run those explicit examples:

>>> from hypothesis import given, example, settings, Phase, strategies as st
>>> @settings(phases=(Phase.explicit,))
... @given(x=st.integers(), y=st.integers())
... @example(x=0, y=0)
... @example(x=0, y=1)
... @example(x=0, y=2)
... @example(x=1, y=0)
... @example(x=1, y=1)
... @example(x=1, y=2)
... def test(x, y):
...   print(x, y)
...   assert True
... 
>>> test()
0 0
0 1
0 2
1 0
1 1
1 2

Also note that st.just(0) | st.just(1) is equivalent to st.one_of(st.just(0), st.just(1)) so choose an approach and stick to it, but don't mix them.

Upvotes: 3

Related Questions