Why does this unit test pass on my machine but fail on the build server?

Question

I am using VS2010, writing unit tests with MSTest. My project uses WPF, MVVM and the PRISM framework. I am also using Moq to mock interfaces.

I am testing the interaction between a command and a selected item in a list. The interaction is encapsulated in a ViewModel according to the MVVM pattern. Basically, when the SelectedDatabase is set, I want the Command to raise CanExecute. I have written this test for the behaviour:

public void Test()
{
    var databaseService = new Mock<IDatabaseService>();
    var databaseFunctionsController = new Mock<IDatabaseFunctionsController>();

    // Create the view model
    OpenDatabaseViewModel viewModel
         = new OpenDatabaseViewModel(databaseService.Object, databaseFunctionsController.Object);

    // Mock up the database and its view model
    var database = TestHelpers.HelpGetMockIDatabase();
    var databaseViewModel = new DatabaseViewModel(database.Object);

    // Hook up the can execute changed event
    var resetEvent = new AutoResetEvent(false);
    bool canExecuteChanged = false;
    viewModel.OpenDatabaseCommand.CanExecuteChanged += (s, e) =>
        {
             resetEvent.Set();
             canExecuteChanged = true;
        };

    // Set the selected database
    viewModel.SelectedDatabase = databaseViewModel;

    // Allow the event to happen
    resetEvent.WaitOne(250);

    // Check that it worked
    Assert.IsTrue(canExecuteChanged,
        "OpenDatabaseCommand.CanExecuteChanged should be raised when SelectedDatabase is set");
}

On the OpenDatabaseViewModel, the SelectDatabase property is as follows:

    public DatabaseViewModel SelectedDatabase
    {
        get { return _selectedDatabase; }
        set
        {
            _selectedDatabase = value;
            RaisePropertyChanged("SelectedDatabase");
            // Update the can execute flag based on the save
            ((DelegateCommand)OpenDatabaseCommand).RaiseCanExecuteChanged();
        }
    }

And also on the viewmodel:

    bool OpenDatabaseCanExecute()
    {
        return _selectedDatabase != null;
    }

TestHelpers.HelpGetMockIDatabase() just gets a mock IDatabase with some properties set.

This test passes when I run the test from VS2010, but fails when executed as part of an automated build on the server. I put in the AutoResetEvent to try to fix the problem, but it's had no effect.

I discovered that the automated tests were using the noisolation flag in the MSTest command line, so I removed that. However, that produced a 'pass' once, but a 'fail' the next.

I think I am missing something important in all of this, but I can't figure out what it is. Can anyone help by telling me what I'm doing wrong?

Answer 1

I have found an answer to explain what was going on with this unit test. There were other complicating factors that I didn't realise were significant at the time. I didn't include these details in my original question because I did not think they were relevant.

The view model described in the question of code is part of a project that is using integration with WinForms. I am hosting a PRISM shell as a child of an ElementHost. Following the answer to the question on stackoverflow How to use Prism within an ElementHost, this is added to create an appropriate Application.Current:

public class MyApp : System.Windows.Application
{
}

if (System.Windows.Application.Current == null)
{
    // create the Application object
    new MyApp();
}

The above code is not exercised by the unit test in question. However, it was being exercised in other unit tests that were being run beforehand, and all were run together using the /noisolation flag with MSTest.exe.

Why should this matter? Well, buried in the PRISM code that is called as a consequence of

((DelegateCommand)OpenDatabaseCommand).RaiseCanExecuteChanged();

in the internal class Microsoft.Practices.Prism.Commands.WeakEventHandler is this method:

public static DispatcherProxy CreateDispatcher()
{
    DispatcherProxy proxy = null;
#if SILVERLIGHT
    if (Deployment.Current == null)
        return null;

    proxy = new DispatcherProxy(Deployment.Current.Dispatcher);
#else
    if (Application.Current == null)
        return null;

    proxy = new DispatcherProxy(Application.Current.Dispatcher);
#endif
    return proxy;

}

It then uses the dispatcher to call the event handler in question:

private static void CallHandler(object sender, EventHandler eventHandler)
{
    DispatcherProxy dispatcher = DispatcherProxy.CreateDispatcher();

    if (eventHandler != null)
    {
        if (dispatcher != null && !dispatcher.CheckAccess())
        {
            dispatcher.BeginInvoke((Action<object, EventHandler>)CallHandler, sender, eventHandler);
        }
        else
        {
            eventHandler(sender, EventArgs.Empty);
        }
    }
}

So it attempts to dispatch the event on the UI thread on the current application if there is one. Otherwise it just calls the eventHandler. For the unit test in question, this led to the event being lost.

After trying many different things, the solution I settled on was just to split up the unit tests into different batches, so the unit test above is run with Application.Current == null.

Answer 2

Updated

The only other remaining places where your code could fail is in these two lines in your snippet for the SelectedDatabase property.

        RaisePropertyChanged("SelectedDatabase");
        // Update the can execute flag based on the save
        ((DelegateCommand)OpenDatabaseCommand).RaiseCanExecuteChanged();

There are others who have had some problems with RaisePropertyChanged() and it's use of magic strings; but this is probably not your immediate problem. Nonetheless, you can look at these links if you want to go down the path of removing the magic string dependency.

WPF, MVVM, and RaisePropertyChanged @ WilberBeast
MVVM - RaisePropertyChanged turning code into a mess

The RaiseCanExecuteChanged() method is the other suspect, and looking up documentation in PRISM reveals that this method expects to dispatch events on the UI thread. From mstest, there are no guarantees that a UI thread is being used to dispatch tests.

DelegateCommandBase.RaiseCanExecuteChanged @ MSDN

I recommend you add a try/catch block around it and see if any exceptions are thrown when RaiseCanExecuteChanged() is called. Note the exceptions thrown so that you can decide how to proceed next. If you absolutely need to test this event dispatch, you may consider writing a tiny WPF-aware app (or perhaps a STAThread console app) that runs the actual test and exits, and having your test launch that app to observe the result. This will isolate your test from any threading concerns that could be caused by mstest or your build server.

Original

This snippet of code seems suspect. If your event fires from another thread, the original thread may exit the wait first before your assignment, causing your flag to be read with a stale value.

viewModel.OpenDatabaseCommand.CanExecuteChanged += (s, e) =>
    {
         resetEvent.Set();
         canExecuteChanged = true;
    };

Consider reordering the lines in the block to this:

viewModel.OpenDatabaseCommand.CanExecuteChanged += (s, e) =>
    {
         canExecuteChanged = true;
         resetEvent.Set();
    };

Another issue is that you don't check if your wait was satisfied. If 250ms did elapse without a signal, your flag will be false.

See WaitHandle.WaitOne to check what return values you'll receive and update this section of code to handle the case of an unsignaled exit.

// Allow the event to happen
resetEvent.WaitOne(250);

// Check that it worked
Assert.IsTrue(canExecuteChanged,
    "OpenDatabaseCommand.CanExecuteChanged should be raised when SelectedDatabase is set");