CODEWITHSUNDEEP
azure-devopsazure-devops-pipelines
kiewic
kiewic

Reputation: 16430

Visual Studio Test task errors ignored after "Failed tests if any were marked as flaky hence changing the exit code to 0"

I have a pipeline with a "Visual Studio Test" task where "Minimum # of tests" is set to 10:

test task minimum number of tests configuration

However, when a release does not meet the 10 minimum tests requirements, the right error is included in the logs:

2021-02-18T04:47:15.4609022Z ##[error]The specified minimum number of tests 10 were not executed in the test run.

azure devops test results

However, there two lines below, there us the following log:

2021-02-18T04:47:16.2005970Z Execution Result Code 1 is non zero, checking for failed results
2021-02-18T04:47:17.8073253Z Failed tests if any were marked as flaky hence changing the exit code to 0

And the task is marked as successful. None of the tests in the entire test suite is marked as flaky? Where can the Failed tests if any were marked as flaky hence changing the exit code to 0 log coming from?

agent job logs

Upvotes: 0

Views: 503

Answers (1)

Hugh Lin
Hugh Lin

Reputation: 19411

Failed tests if any were marked as flaky hence changing the exit code to 0

You can try to disable Flaky test detection in the Test management of project settings.

enter image description here

Flaky tests present a barrier to finding real problems, since the failures often don't relate to the changes being tested. A flaky test is a test that provides different outcomes, such as pass or fail, even when there are no changes in the source code or execution environment. Flaky tests also impact the quality of shipped code.

For details , please refer to this document.

Upvotes: 1

Related Questions