Unittest strategy for a function performing an experiment that includes some randomness

Question

What approach should I take to write a unittest for this function?

Please note that:

• at each execution this function will generate different list of results for the same input parameters (with a very high probability).
• the list might be empty at the end of the execution (if we have reached out the maximum number of tries without finding a single result that is seen as valid)
• NUMBER_OF_RESULTS and MAX_TRIES are > 0 and MAX_TRIES is way larger than NUMBER_OF_RESULTS

def perform_experiment(some parameters) -> results[obj]:
  results = []
  for i in range(MAX_TRIES):
    result_to_validate = random_attempt()
    if valid(result_to_validate):
      results.append(result_to_validate)
    if len(results) >= NUMBER_OF_RESULTS:
      break
  return results

I was thinking of implementing in the unittest in the following way

1. When the list of results is NOT empty, then I can simply go through all the elements and assert each of them are valid. Which isn't difficult to write.
2. If the result list is empty, I would like to make sure that the perform_experiment has run until i has reached MAX_TRIES, however the variable i is not accessible outside of the function.

I am not sure how I could test the 2. point in a unittest, should I change this into making sure that the function to test has run at least for a certain amount of time instead of checking that i has reached the MAX_TRIES threshold? Is using a seed the only option here? What can be done if we can't use one? Or can we completely omit point 2. from the unittest?

Answer 1

First, you need to understand the context you are in. So it is a function you should test. But it is not a real function, where input goes in and output goes out with some processing inside: this function has some external dependecies which is always something disturbing and makes testing harder. So, refactor it:

def perform_experiment(maxTries, numberOfResults, some parameters) -> results[obj]:

Now, there is still 1 dependency we cannot control: random_attempt(). This is the reason why you have to mock it. Let's move it to separate function to increase the clarity and make mocking easier:

def perform_experiment(maxTries, numberOfResults, some parameters) -> results[obj]:
  results = []
  for i in range(maxTries):
    results.append(getValidRandomAttempt())
    if len(results) >= numberOfResults:
      break
  return results

def getValidRandomAttempt():
   result_to_validate = random_attempt()
      if valid(result_to_validate):
        return result_to_validate

But wait, we want to just collect valid results so we have to change list name, also there is problem with None value when no valid attempt was made:

def perform_experiment(maxTries, numberOfResults, some parameters) -> validResults[obj]:
  validResults = []
  for i in range(maxTries):
    result = getValidRandomAttempt()
    if result != None:
      validResults.append(result)
    if len(validResults) >= numberOfResults:
      break
  return validResults


def getValidRandomAttempt():
   result_to_validate = random_attempt()
      if valid(result_to_validate):
        return result_to_validate

At this stage there is the hardest part as you have to understand the expectations about this function given the maxTries, numberOfResults and results list values combination. This is the example which may be too detailed or may be missing some aspects. This input model needs to be tuned to your needs. Anyway:

There are 2 cases about the maxTries:

A. maxTries == 0
B. maxTries > 0

Then, there are cases when maxTries > 0 about maxTries to numberOfResults relation:

A. maxTries > numberOfResults
B. maxTries == numberOfResults
C. maxTries < numberOfResults

Finally, they should be joined with validResults cases:

0. maxTries == 0
1. maxTries > numberOfResults, len(validResults) == numberOfResults
2. maxTries > numberOfResults, len(validResults) > numberOfResults
3. maxTries > numberOfResults, len(validResults) < numberOfResults
4. maxTries == numberOfResults, len(validResults) == numberOfResults
5. maxTries == numberOfResults, len(validResults) > numberOfResults
6. maxTries == numberOfResults, len(validResults) < numberOfResults
7. maxTries < numberOfResults, len(validResults) == numberOfResults
8. maxTries < numberOfResults, len(validResults) > numberOfResults
9. maxTries < numberOfResults, len(validResults) < numberOfResults

We can try to add expectations now and create testcases with mocking getValidRandomAttempt appropriately:

0. GIVEN
mock_getValidRandomAttempt.side_effect = [None]
WHEN
resultList = perform_experiment(0, 0, some parameters)
THEN
assert len(resultList)==0

1. GIVEN
mock_getValidRandomAttempt.side_effect = [None, 1, 1, 1]
WHEN
resultList = perform_experiment(5, 3, some parameters)
THEN
assert len(resultList)==3

2. GIVEN
mock_getValidRandomAttempt.side_effect = [1, 1, 1, 1]
WHEN
resultList = perform_experiment(5, 3, some parameters)
THEN
assert len(resultList)==3

3. GIVEN
mock_getValidRandomAttempt.side_effect = [1, None, None, None, None]
WHEN
resultList = perform_experiment(5, 3, some parameters)
THEN
assert len(resultList)==1

4. GIVEN
mock_getValidRandomAttempt.side_effect = [1, 1, 1, 1, 1]
WHEN
resultList = perform_experiment(5, 5, some parameters)
THEN
assert len(resultList)==1

5. GIVEN
mock_getValidRandomAttempt.side_effect = [1, 1, 1, 1, 1, 1]
WHEN
resultList = perform_experiment(5, 5, some parameters)
THEN
assert len(resultList)==5

Last but not least I encourage you not to use Python if this is possible. It is amazing how popular it is, while having very serious drawbacks when testing: very unclear mocking syntax with tons of decorators which render tests almost completely unreadable.