TFS2015 Test Agent Browser.Launch generating .NullReferenceException: Object reference not set to an instance of an object

Question

We are getting a NullReference Exception from Browser.Launch method in our codedUI running under TFS2015 agent. Oddly, the IE browser (version:11.0.9600.18376) loads on the test agent, navigated to the correct web app and then shuts down immediately.

We have setup an on-premise TFS 2015 update 3 environment with a build server and a separate test agent machine. The codedUI project has been upgraded from Visual Studio 2013 to Visual Studio 2015. We changed the project references for Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Microsoft.VisualStudio.QualityTools.UITest.Common, Microsoft.VisualStudio.TestTools.UITest.Extension, and Microsoft.VisualStudio.TestTools.UITesting to use assemblies from Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\PublicAssemblies\

We create a TFS2015 build, deploy and test for codedUI solution using an agent configuration of Interactive Process and the Run Functional Tests build step.

Our codedUI follows a simple pattern of driving a browser through a web application. The code to launch is

BrowserWindow.CurrentBrowser = "ie";
BrowserWindow browserWindow = BrowserWindow.Launch(new System.Uri("https://app.web.com"));

It is the call to BrowserWindow.Launch method that is coming back with a nullreference exception. Here is a stack trace:

at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.GetPropertyValuePrivate(String propertyName)
at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.GetPropertyPrivate(String propertyName)
at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.GetPropertyOfType[T](String propertyName)
at Microsoft.VisualStudio.TestTools.UITesting.UITestControl.get_WindowHandle()
at Microsoft.VisualStudio.TestTools.UITesting.BrowserWindow.SwitchToWindow(BrowserWindow browserWindow)
at Microsoft.VisualStudio.TestTools.UITesting.BrowserWindow.LaunchPrivate(Uri uri)
at Microsoft.VisualStudio.TestTools.UITesting.BrowserWindow.<>c__DisplayClass2_0.<Launch>b__0()
at Microsoft.VisualStudio.TestTools.UITesting.CodedUITestMethodInvoker.InvokeMethod[T](Func`1 function, UITestControl control, Boolean firePlaybackErrorEvent, Boolean logAsAction)
at Microsoft.VisualStudio.TestTools.UITesting.BrowserWindow.Launch(Uri uri)
at DPOCommon.Case.launchBrowser(String browser, String URL) in D:\agent\agent\_work\1\s\DPORegression\DPOCommon\Case.cs:line 136

Error Message:

Test method ScreenDesigner.TC407.TC407TestMethod_IE threw exception: System.NullReferenceException: Object reference not set to an instance of an object. 
TestCleanup method ScreenDesigner.TC407.TestCleanup threw exception. System.NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object..

Added info: We are not using the old XAML build. We are using the new web based build definition. The steps are Nuget Installer, Visual Studio Build, Windows Machine File Copy, Visual Studio Test Agent Deployment, and Run Functional Test. Each step is working except that the Run Functional Test step executes the codedUI test, brings up the IE browser on the test agent machine, and then gets this error. Following that the browser shuts down and repeats the same sequence with all remaining tests in the test suites. For the Run Functional Test, we are using the execution options of: TestSelection=Test Plan with a test plan selected and test suite selected. Our test configuration is set to windows 7 and IE. We also have the run settings files pointed to its location in our source control. Code Coverage is disabled. Within the test settings configuration we only have screen and video recording enabled. These both seem to work since we see the failed test result with a screen and video recording of the brief loading of IE.

Also, we tried to diagnose this issue by copying the codedUI project and source code to the test agent manually and then installing Visual Studio 2015 directly on the test agent machine. When we load the codedUI project within Visual Studio 2015 on the test agent machine and run the codedUI test it works successfully. However, the same codedUI remains not working with the "null object reference" issue when run through the "run functional tests" automated build step.

update 08/11/2016: Adding our test.testsettings file. After further diagnose to respond to comment question, we discovered that when we remove the test.testsettings file from the "Run Functional Tests" steps then the codeUI tests run without getting an object null reference error after loading the browser. Unfortunately, we need to use the legacy test.testsettings file because we need to include screen recording data collector so that failed tests show the web application sequence leading up to the error for our development team to diagnose

<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="TestSettings" id="caa2702e-3ce5-4b20-be15-5e85ef063642" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description>These are default test settings for a local test run.</Description>
  <Deployment enabled="false" />
  <Execution>
    <TestTypeSpecific>
      <UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
        <AssemblyResolution>
          <TestDirectory useLoadContext="true" />
        </AssemblyResolution>
      </UnitTestRunConfig>
      <WebTestRunConfiguration testTypeId="4e7599fa-5ecb-43e9-a887-cd63cf72d207">
        <Browser name="Internet Explorer 9.0" MaxConnections="6">
          <Headers>
            <Header name="User-Agent" value="Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" />
            <Header name="Accept" value="*/*" />
            <Header name="Accept-Language" value="{{$IEAcceptLanguage}}" />
            <Header name="Accept-Encoding" value="GZIP" />
          </Headers>
        </Browser>
      </WebTestRunConfiguration>
    </TestTypeSpecific>
    <AgentRule name="LocalMachineDefaultRole">
      <DataCollectors>
        <DataCollector uri="datacollector://microsoft/VideoRecorder/1.0" assemblyQualifiedName="Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder.VideoRecorderDataCollector, Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" friendlyName="Screen and Voice Recorder">
        </DataCollector>
      </DataCollectors>
    </AgentRule>
  </Execution>
  <Properties />
</TestSettings>

Answer 1

With @s09 diagnostic help, we have found a solution to this issue. I was able to successful run the build and "Run Functional Tests" through the vNext Build when I configured our environment to run the build and codedUI on the same machine (Windows Server 2012). In doing this, I had to install Microsoft Expression Encoder 4 and set the Desktop experience per these instructions:https://msdn.microsoft.com/en-us/library/dd997558(v=vs.100).aspx

So now that we hade a working build scenario I need to get this working on our distributed test agent machine. Running codedUI on our build server was not a feasible solution. So I applied the follow steps to the test agent machine, which is separate from our build server and is running on Windows 7:

1. Applied Expression Encoder 4.0 to the test agent machine
2. Cloned our build definition so that I could work with independent of the scheduled build definition Added our test.testsettings to the "Run Functional Tests" task. This includes a data collection for screen recording.
3. Ran the build and the codedUI test work successfully without the null reference error

Hmmm, ok now lets try the original build definition by adding the same test.settings file which is also compiled the same codedUI project and running the same test plan->test suite. Kaboom, this run failed with the null reference error. Ok now that I had one working build definition and one non-working build definition of the same project, etc, I looked at the compiled binary results kept within the build servers working folders. In my case this was under D:\agent\agent_work (on the build server) where there as numeric folder for each respective build definition. The compare revealed that there were as set of extraneous assemblies in the build binary folder that should not have been there. They were most likely placed in the folder during an early build cycle as we migrated our project from VS2013 to VS2015 and had to adjust the project references to use V14.0 of these assemblies. And we removed the copy to local directory for these assemblies.

Original Build binary folder showing extra assemblies

Cloned Build binary folder

So I deleted the entire working folder directory for the original build definition from the build server. Queued up a new build and Bingo...the codedUI tests ran on the distributed test agent machine successfully.

Thank you again @s09 for your invaluable help in diagnosing this issue.