CODEWITHSUNDEEP
.nettfsmstest
Vincent Hubert
Vincent Hubert

Reputation: 1396

All tests pass but TFS marks the build as partially succeded

Our current project involves building a robotized box controlled by a .Net application. We interface with quite a few hardware libraries and we did set up a integration server with all the hardware connected to it to run nightly regression tests.

Unfortunately, not all hardware libraires in our setup integrates nicely with TFS and MSTest.

When we run our build and tests with a certain librairy, we have either one of these 2 behaviors:

Looking at the logs, we can see these 2 lines after the tests are published to the TFS server, "Handle MSTest Exception" "MSTest.exe returned an exit code of 0 indicating that not all tests passed"

What really puzzles me is that running the same tests with mstest from the command line does not display this problem. Futhermore, when running from the command line, MsTest returns 0 when all tests pass and 1 when there is an error

So my questions are:

  1. What is the appropriate MSTest return code on Success/Failure
  2. Beside the log in visual studio, is there any other detailed loggin feature available?
  3. Any clue on where to look?

Additional info: We did notice a difference when using (or not) the "NoIsolation" flag in mstest. The specific tests will pass when using that flag, however other tests will fail... We are still investigating that one.

Thanks

EDIT: The relevant portion of the log: 30/30 test(s) Passed

Summary
-------
Test Run Completed.
Passed  30
----------
Total   30
Results file:  C:\Builds\1\Galil Daily build\TestResults\D201364-W7$_D201364-W7 2011-07-05 10_23_33_Any CPU_Debug.trx
Test Settings: Default Test Settings
Waiting to publish...
Publishing results of test run D201364-W7$@D201364-W7 2011-07-05 10:23:33_Any CPU_Debug to http://mtlapp07:8080/tfs/DI_DEV...
..Publish completed successfully.
Final Property Values
Category = Galil
CommandLineArguments = /noisolation
Flavor = 
MaxPriority = -1
MinPriority = -1
PathToResultsFilesRoot = C:\Builds\1\Galil Daily build\TestResults
Platform = 
Publish = True
SearchPathRoot = C:\Builds\1\Galil Daily build\Binaries
TestConfigId = -1
TestConfigName = 
TestContainers = System.Linq.OrderedEnumerable`2[System.String,System.String]
TestLists = 
TestMetadata = 
TestNames = 
TestSettings = 
ToolPath = 
Version = -1
Final Property Values
Condition = False
Final Property Values
Condition = True

00:00
Handle MSTest Exception
 MSTest.exe returned an exit code of 0 indicating that not all tests passed.

00:00
If testException is NOT TestFailureException
Initial Property Values
Condition = False
Final Property Values
Condition = False

Edit 2.0:

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Galil;

namespace TestProject1
{
[TestClass]
public class UnitTest1
  {
    [TestMethod]
    public void TestMethod1()
    {
        Galil.Galil m_galil = new Galil.Galil();
        m_galil.address = "COM4 19200";
        string resp = m_galil.command("MTA=2.5;");
        Assert.AreEqual(":", resp);
        try
        {
            m_galil.address = "OFFLINE";
        }
        catch (Exception)
        {
        }
    }

    [TestMethod]
    public void TestMethod2()
    {
        Galil.Galil m_galil = new Galil.Galil();
        m_galil.address = "COM4 19200";
        string resp = m_galil.command("MTA=2.0;");
        Assert.AreEqual(":", resp);
        try
        {
            m_galil.address = "OFFLINE";
        }
        catch (Exception)
        {
        }
    }
  }
}

Upvotes: 2

Views: 2898

Answers (2)

Vincent Hubert
Vincent Hubert

Reputation: 1396

Actually, we were able to isolate the problem to a library that did output strings into the standard error stream. Somehow, if TFS sees something on the standard error, it will flag the test as partially completed.

Upvotes: 2

Mike Veigel
Mike Veigel

Reputation: 3815

What is the appropriate MSTest return code on Success/Failure

Exit code 1 = not all tests pass

Example:

C:\Program Files\MSBuild\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets(1377,5,1377,5): warning : MSTest.exe returned an exit code of 1 indicating that not all tests passed.

All tests pass

Example:

Waiting to publish... Publishing results of test run Company_TFSBuild_SVC@BUILD-DEV 2011-07-01 13:15:46_Any CPU_Release to http://company-source:8080/...
..Publish completed successfully.

Beside the log in visual studio, is there any other detailed loggin feature available?

You can get a more detailed log by going to:

Menu Bar -> Tools -> Options... -> Projects and Solutions -> MSBuild project build output verbosity

Any clue on where to look?

Need to see your log file.

Upvotes: 2

Related Questions