Can't run specflow tests from jenkins and `runtests.cmd` file

Question

I'm trying to run specflow tests from jenkins but all the tests are running expect specflow tests (spec flow tests should be fail - assert 1==2).

Note: Using visual studio 2017, all the tests including spec flow are running as expected.

Jenkins configuration: (no errors but with warnings)

Jenkins console output:

...
Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.17
Path To MSTest.exe: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\MSTest.exe
Delete old result file file:/C:/Program%20Files%20(x86)/Jenkins/workspace/new%203/TestResults.trx
[new 3] $ "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\MSTest.exe" /resultsfile:TestResults.trx /noisolation "/testcontainer:C:\Program Files (x86)\Jenkins\workspace\new 3\WebApplication2.Tests\bin\Debug\WebApplication2.Tests.dll"
Microsoft (R) Test Execution Command Line Tool Version 15.0.26621.2
Copyright (c) Microsoft Corporation. All rights reserved.

Loading C:\Program Files (x86)\Jenkins\workspace\new 3\WebApplication2.Tests\bin\Debug\WebApplication2.Tests.dll...
Starting execution...
Results               Top Level Tests
-------               ---------------
Passed                WebApplication2.Tests.Controllers.HomeControllerTest.About
Passed                WebApplication2.Tests.Controllers.HomeControllerTest.Contact
Passed                WebApplication2.Tests.Controllers.HomeControllerTest.Index
3/3 test(s) Passed

Summary
-------
Test Run Warning.
  Passed  3
  ---------
  Total   3
Results file:  C:\Program Files (x86)\Jenkins\workspace\new 3\TestResults.trx
Test Settings: Default Test Settings

Run has the following issue(s):
Warning: Test Run deployment issue: The assembly or module 'WebGrease' directly or indirectly referenced by the test container 'c:\program files (x86)\jenkins\workspace\new 3\webapplication2.tests\bin\debug\webapplication2.tests.dll' was not found.
[MSTEST-PLUGIN] INFO processing test results in file(s) TestResults.trx
[MSTEST-PLUGIN] INFO processing report file: C:\Program Files (x86)\Jenkins\workspace\new 3\TestResults.trx
[MSTEST-PLUGIN] WARNING Unable to delete the file C:\Program Files (x86)\Jenkins\workspace\new 3\temporary-junit-reports
[MSTEST-PLUGIN] INFO This file is a reserved temporary file. You can delete it safely.
[MSTEST-PLUGIN] INFO XML coverage report file not found: C:\Program Files (x86)\Jenkins\workspace\new 3\vstest.coveragexml

[MSTEST-PLUGIN] INFO XML coverage report file not found: C:\Program Files (x86)\Jenkins\workspace\new 3\TestResults.coveragexml

Finished: SUCCESS

When I run runtests.cmd there are multiple errors:

C:\Users\stava\source\repos\WebApplication2\WebApplication2.Tests>C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe "" 
Microsoft (R) Build Engine version 4.7.2053.0
[Microsoft .NET Framework, version 4.0.30319.42000]
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 05/10/2017 23:38:25.
Project "C:\Users\stava\source\repos\WebApplication2\WebApplication2.Tests\WebApplication2.Tests.csproj" on node 1 (default targets).
Project file contains ToolsVersion="15.0". This toolset may be unknown or missing, in which case you may be able to resolve this by installing the appropriate version of MSBuild, or the build may have been forced to a particular ToolsVersion for policy reasons. Treating the project as if it had ToolsVersion="4.0". For more information, please see http://go.microsoft.com/fwlink/?LinkId=291333.
Project "C:\Users\stava\source\repos\WebApplication2\WebApplication2.Tests\WebApplication2.Tests.csproj" (1) is building "C:\Users\stava\source\repos\WebApplication2\WebApplication2\WebApplication2.csproj" (2) on node 1 (default targets).
C:\Users\stava\source\repos\WebApplication2\WebApplication2\WebApplication2.csproj(229,3): error MSB4019: The imported project "C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\WebApplications\Microsoft.WebApplication.targets" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.
Done Building Project "C:\Users\stava\source\repos\WebApplication2\WebApplication2\WebApplication2.csproj" (default targets) -- FAILED.
Done Building Project "C:\Users\stava\source\repos\WebApplication2\WebApplication2.Tests\WebApplication2.Tests.csproj" (default targets) -- FAILED.

Build FAILED.

"C:\Users\stava\source\repos\WebApplication2\WebApplication2.Tests\WebApplication2.Tests.csproj" (default target) (1) ->
"C:\Users\stava\source\repos\WebApplication2\WebApplication2\WebApplication2.csproj" (default target) (2) ->
  C:\Users\stava\source\repos\WebApplication2\WebApplication2\WebApplication2.csproj(229,3): error MSB4019: The imported project "C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\WebApplications\Microsoft.WebApplication.targets" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.

    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:00.19

Why the tests run prefectly fine form VS and fail to run from runtests.cmd? Is this related to why the tests don't run in jenkins?

Answer 1

I assume you are using the SpecFlow+Runner because you have the runtests.cmd.
In that case, please have a look at the documentation how to configure Jenkins with it: http://specflow.org/plus/documentation/SpecFlowPlus-and-Jenkins/

You have to use vstest.console.exe and not mstest.exe MSTest.exe is the runner when you are using MSTest as test runner.

If you are not using the SpecFlow+Runner, please look in the documentation for your unit test runner.