CODEWITHSUNDEEP
unit-testinglanguage-agnosticphpunit
ellabeauty
ellabeauty

Reputation: 2599

Should all unit tests be successful?

Let's say you make an application that attempts to transliterate stuff from alphabet A into alphabet B, as closely as possible.

Because language B is very complex, this is not always successful. But you do get an approximate transliteration.

How would you build unit tests in this case, considering that you expect 20-30% to fail?

Upvotes: 6

Views: 427

Answers (5)

Michel Keijzers
Michel Keijzers

Reputation: 15367

The only reason a unit test might fail is if timing is involved (mostly in case of electronics being tested). However, even if those cases, it should be the goal to remove the timing problem from the unit test, e.g. by extending/switching of the timeout or other timing issue if possible. When it is not possible, it should be well documented.

Another way to remove timing issues and make tests deterministic, is to write stubs for all external interfaces, with some injection method, i.e. be able to set values that external interface methods will return. By setting up a unit test this way, you can literally test everything an also every error condition.

(story: I worked at a company where several unit tests would fail occasionaly. Only a few people were able to analyze if those were serious errors or a timing issue. It will save a lot time to make good unit tests in the first place).

Upvotes: 1

Lynn Langit
Lynn Langit

Reputation: 4060

A technique that I've learned in this situation is to test only the parts of the code that are functional (or deterministic). Of course, the difficult part if to separate out the deterministic from the non-deterministic parts. The shorthand way to say this is to 'reduce [or refactor] to functional', which means to separate out the parts of the code that are deterministic, and then test those parts.

For a scenario-based bit of context, read this blog post about the application of this technique when getting tests around legacy code (and using the open source unit testing library named ApprovalTests).

Yet another technique that may be of interest here is 'theory-based' testing. For more on that, check out this blog post.

Upvotes: 3

Marcel Hebing
Marcel Hebing

Reputation: 3182

It must always be the goal that your unit test is successful. The way you use it, you can not differentiate between serious errors in your software and the errors in the translations you seem to expect.

I would suggest to separate both:

  1. Use unit tests only to make sure that your software works the way you expect it (even if this includes some mistakes in the translations)
  2. Produce a test dataset containing the translations produced by your software. You can check this test dataset either manually or using a statistical software package like R to get an idea of the quality of your translations.

Upvotes: 8

k.m
k.m

Reputation: 31484

Unit test should be deterministic. Failing test should indicate software failure, not something that "works as intended". For your case, prepare data in a way you can be sure of results and test for them, whether conversion succeeds or fails (testing for failure is always an option, given you expect failure - important is that you're always in control of when your test passes/fails).

Upvotes: 7

amdn
amdn

Reputation: 11582

A unit test that is expected to fail is not a unit test. You need to change the definition of success by using an evaluation function that acts as a filter and decides if it is "close enough" and determines pass/fail. As your translator gets better, you can raise the bar in the filter.

Upvotes: 5

Related Questions