CODEWITHSUNDEEP
unit-testingtfsbuild
dwilliss
dwilliss

Reputation: 924

TFS Build servers and critical Unit Tests

When you build on a TFS build server, failed unit tests cause the build to show an orange alert state but they still "succeed". Is there any way to tag a unit test as critical such that if it fails, the whole build will fail?

I've Googled for it and didn't find anything, and I don't see any attribute in the framework, so I'm guessing the answer is no. But maybe I'm just looking in the wrong place.

Upvotes: 1

Views: 473

Answers (2)

jessehouwing
jessehouwing

Reputation: 114491

There is a way to do this, but you need to create multiple test runs and then filter your tests. On your tests, set a TestCategory attribute:

[TestCategory("Critical")]
[TestMethod]
public void MyCriticalTest {}

For NUnit you should be able to use [Category("Critical")]. There are multiple attributes of a test you can filter on, including the name.

  • Name = TestMethodDisplayNameName
  • FullyQualifiedName = FullyQualifiedTestMethodName
  • Priority = PriorityAttributeValue
  • TestCategory = TestCategoryAttributeValue
  • ClassName = ClassName

And these operators:

  • = (equals)
  • != (not equals)
  • ~ (contains or substring only for string values)
  • & (and)
  • | (or)
  • ( ) (paranthesis for grouping)

XUnit .NET currently does not support TestCaseFilters.

Then in your build definition you can create two test runs, one that runs Critical tests, one that runs everything else. You can use the Filter option of the Test Run.

Open the Test Runs window using this hard to find button: enter image description here

Create 2 test runs: enter image description here

On your first run set the options as follows: enter image description here

On your second run set the options as follows: enter image description here

This way Team Build will run any test with the "Ciritical" category in the first run and will fail. If the first run succeeds it will kick off the non-critical tests and will Partially Succeed, even when a test fails.

Update

The same process explained for Azure DevOps Pipelines.

Upvotes: 5

Daniel Mann
Daniel Mann

Reputation: 58980

Yes.

Using the TFS2013 Default Template:

Under the "Process" tab, go to section 2, "Basic".

Expand the Automated Tests section.

For "Test Source", click the ellipsis ("...").

This will open a new window that has a "Fail build when tests fail" check box.

Upvotes: 0

Related Questions