CODEWITHSUNDEEP
pythonpytestpython-unittest
aleskva
aleskva

Reputation: 1805

Allow failure for all tests if specific Error is thrown

I have more than 2000 unittests/pytests in my project. Many of them ask API, but API can be lagged. Is there a way to expect APILagError in every single test and throw SKIP/XFAIL for any test if this error occurs?

Current:

result: 1 fail, 2 ok => tests failed

Wanted:

result: 2 ok, 1 skipped/xfailed => tests passed

Upvotes: 0

Views: 732

Answers (1)

Chris
Chris

Reputation: 7278

I feel your pain. I'm going to do that annoying SO thing where I don't actually answer your question as you posed it, but suggest you change directions. You should choose one of:

  1. mock that API so it doesn't time out any more and your tests aren't flakey
  2. treat your API lag as normal behavior and increase the allowed duration of API calls so typical lag time doesn't result in a failed test
  3. treat your API lag as a bug and go fix it so your tests pass

Why are flakey tests bad?

Flakey tests mean you can't tell the difference between broken code and a slow API call. So your tests aren't helping you catch bugs. So what's the point?

Why would I ever mock the API? It's not "real"!

You'd do this if you want to test anything outside of the API. Decouple your logic from the API's behavior to eliminate flakiness and make your code more maintainable.

You might also be able to get a similarly-behaved but much faster API to test against as your mock. Like check out the API's code locally and populate it with a small set of data. Hit that in your tests instead of the production API. That would let you check its logic separate from its lagginess.

Upvotes: 3

Related Questions