CODEWITHSUNDEEP
azure-devopsazure-pipelinesreleasespecflow
AutoTester213
AutoTester213

Reputation: 2862

Azure Devops - Release Pipeline when re-running failed tests azure devops shows failure status even if re-run succeeded

I use Specflow with SpecRunner+ I am using the Deafult.srprofile to to re-run failed tests 3 times in visual studio it shows 2passed 1 failed but the status of the test is a failure, the same goes for azure devops if a re-ran test passes the outcome of the run is a failure. The Failures are sometimes caused by locator timeouts or server timeouts not often but saw it happen few time thats why we decided to implement a re-run.

Could anyone help on this?

022-02-09T12:40:13.8607507Z Test Run Failed.
2022-02-09T12:40:13.8608607Z Total tests: 37
2022-02-09T12:40:13.8609271Z      Passed: 36
2022-02-09T12:40:13.8609858Z      Failed: 1
2022-02-09T12:40:13.8617476Z  Total time: 7.4559 Minutes
2022-02-09T12:40:13.9226929Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2022-02-09T12:40:14.0075402Z ##[error]Error: The process 'D:\Microsoft_Visual_Studio\2019\Common7\IDE\Extensions\TestPlatform\vstest.console.exe' failed with exit code 1
2022-02-09T12:40:14.8164576Z ##[error]VsTest task failed.

But then the report states that it was retried 3 times which 2 of the retries were seccusefull but still a failure status on the azure devops run. Report Result

Upvotes: 0

Views: 1459

Answers (1)

Andreas Willich
Andreas Willich

Reputation: 5825

The behavior of the report is the correct one and sadly this can't be configured to be changed.

What you can do is to adjust how the results are reported back to Azure DevOps. You can configure it via the VSTest element in the srProfile- File.

This example means, that at least one retry has to be passing:

<VSTest testRetryResults="Unified" passRateAbsolute="1"/>

Docs: https://docs.specflow.org/projects/specflow-runner/en/latest/Profile/VSTest.html

Be aware that we have stopped the development of the SpecFlow+ Runner. More details here: https://specflow.org/using-specflow/the-retirement-of-specflow-runner/

Upvotes: 2

Related Questions