Not all Specflow Hooks are executed when running tests on vstest.console with NUnit3TestAdapter

Question

We use the visualstudio.com Release Manager to tell a local Build Agent to build and execute a set of SpecFlow tests on the Build Agent-machine, pulling from a GIT-branch.

We're using the Test Assemblies-task with a Custom Test Adapter (NUnit3TestAdapter 3.7.0) and point it toward the assembly containing the SpecFlow tests. Other assemblies inside the same bin folder contain SpecFlow-Hooks that execute setup and teardown-hooks.

The problem we're facing is that not all of these Hooks seem to called. The first and last messages in the .trx logs are all from [BeforeScenario] and [AfterScenario] hooks. There are -however- [BeforeTestRun], [AfterTestRun] and [BeforeFeature] hooks in the same assemblies.

I've pasted the (abridged) logs from VSTS below:

Executing vstest.console.exe "DLL CONTAINING TESTS" /TestCaseFilter:"TestCategory=MP2" /Settings:"C:\agent_work\r17\a\TestResults\1_2017-05-31_09-31-50-AM.runsettings" /logger:trx /TestAdapterPath:"C:\agent_work\r17\a\drop\b\packages\NUnit3TestAdapter.3.7.0\tools"

Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1 2017-05-31T07:31:51.1862622Z Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...

Information: NUnit Adapter 3.7.0.0: Test execution started

Information: Running all tests in C:\agent_work\r17\a\drop\b\DLL CONTAINING TESTS

Information: NUnit3TestExecutor converted 472 of 472 NUnit test cases

Then the logging from the tests itself, starting with messages from BeforeScenario.

I'm at a loss as to why the other hooks are never called. Does NUnit3TestAdapter even keep track of a TestRun context? Is it because the Before/AfterTestRun and Before/AfterFeature-hooks are bound to static methods? Or is it possible they ARE executed, but the messages are never passed through to the .trx logs?

Answer 1

It turns out [BeforeTestRun] was being called perfectly fine, but just didn't show up in the logs. The problem was with [AfterTestRun], which turns out to be an old bug in SpecFlow that was never really fixed (?)