Remote execution of CodedUI tests on Virtual machine by Test Controller and Agent

Question

I would like to ask some help regarding the case below.

The goal I would like to achieve is having a Virtual Machine Windows 7 SP1 running on it and having that my colleagues - developers - can run the CodedUI tests run on this machine. In order to that I created the VM on my machine, it is available for everybody. I installed a VS 2013 Test Agent and Controller from here: http://www.microsoft.com/en-us/download/details.aspx?id=40750

I set up both and according to the information they can communicate to each other.

I set up a user for running the tests. This user is member of Administrator and other groups which is described in the MSDN pages. When I start the VM it logs in automatically and I got the same you can see the screenshot below:

Furthermore, when I want to set up the agent in VS 2013 it says that it - VS 2013 - can communicate to the Agent placed on the virtual machine. I can see the Agent and Controller properties in VS. However, it strange for me that, the agent name is displayed like this: *agent_name. I mean asterisk and after the agent name.

After I select the testsettings file in the test settings menu and I say to the VS that please run this test on VS it builds and working on it at least 1-2 minutes and it says "Test skipped". It does not matter whether the test is single unit test or mix of SpecFlow/CodedUI. It fails always.

I found interest things in the eventlog's of that machine where my VS runs. The VM's, where the Agent and the Controller run, are empty.

The description for Event ID 0 from source VSTTExecution cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

(vstest.executionengine.x86.exe, PID 8440, Thread 10) ControllerProxy: Queue test run error: System.Net.Sockets.SocketException (0x80004005): No such host is known

Server stack trace: 
   at System.Net.Dns.GetAddrInfo(String name)
   at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
   at System.Net.Dns.GetHostAddresses(String hostNameOrAddress)
   at System.Runtime.Remoting.Channels.RemoteConnection.CreateNewSocket()
   at System.Runtime.Remoting.Channels.RemoteConnection.GetSocket()
   at System.Runtime.Remoting.Channels.SocketCache.GetSocket(String machinePortAndSid, Boolean openNew)
   at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.SendRequestWithRetry(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream)
   at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
   at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.VisualStudio.TestTools.Common.FileCopyService.get_DeploymentFlags()
   at Microsoft.VisualStudio.TestTools.Controller.ControllerExecution.<StartTestRun>b__1e(FileCopyService fileCopyService)
   at Microsoft.VisualStudio.TestTools.Execution.RemoteObjectContainer`1.InvokeAsRemoteUser(Action`1 invoke)
   at Microsoft.VisualStudio.TestTools.Controller.ControllerExecution.StartTestRun(TestRun testRun, RemoteObjectContainer`1 fileCopyServiceContainer, RemoteObjectContainer`1 defaultListenerContainer)
   at Microsoft.VisualStudio.TestTools.Controller.ControllerObject.QueueTestRun(TestRun testRun, FileCopyService fileCopyService, IControllerEvents eventHandler)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [1]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.VisualStudio.TestTools.Controller.IControllerExecution.QueueTestRun(TestRun testRun, FileCopyService fileCopyService, IControllerEvents eventHandler)
   at Microsoft.VisualStudio.TestTools.TestManagement.ControllerProxy.QueueTestRunWorker(Object state)

the message resource is present but the message is not found in the string/message table

If VS and the controller can communicate each other then why I got the "no such host" error?

I went through a lot of descriptions and error reports here and at Microsoft, but I don't know what to do. I've double-checked the settings many times and everything is ok.

Anybody, any idea what can I do?

Answer 1

Here are some things I have run into for anyone looking for this in the future.

1. Running tests through Visual studio. Install Controller and agent manually. When setting up the controller DO NOT register with team collection. When setting up agent set name of controller server including :6901. The system should add the test user to the system automatically. If you are running on a domain you will probably need to add the VM to the domain.

2. Running tests through MTM. Use MTM to install agents and controllers on machine. Only run tests from MTM as they cannot be run from VS.

3. If you want to run tests using visual studio and MTM you will want to setup two VM's one with each setup.

Notes for running tests with Visual Studio Remotely.

1. Start a remote connection to the server that will be running the tests (a remote connection must be open to run tests). There are workarounds if you use hyper-v. The connection can be hidden behind apps, but cannot be minimized.
2. Due to something on our domain setup I cannot run remote tests unless I run connected to the VPN, working with helpdesk to resolve this issue now. Try this if you have connection issues.
3. The settings to disable the screensaver on agent install do not seem to work, at least it did not work for me on 9 different VMs. I had to edit that and auto login through the registry.
4. You can run tests using vstest.console.exe remotely without using visual studio or MTM, I prefer to run a remote powershell script, but you can use a test setting file as well. https://msdn.microsoft.com/en-us/library/jj155796(v=vs.110).aspx Use the same controller and agent setup as if you would be running from Visual Studio.
5. If the controller is on a different domain make sure to use the fully qualified domain name, ie myserver.companyname.corp

Answer 2

Since I had this problem I collected some experience in this topic. However, I feel that these products of Microsoft are very immature.

What I could achieve is that, I set up the virtual machine I mentioned and I'm able to run CudedUI tests on it executed by Test Manager. In order to have this I had to set up a test agent, which driven by a test controller placed in the company's infrastructure somewhere, on the virtual machine by Test Manager. In this case @Eli El-Tawil you were right and thanks for the suggestion.

However, if I want to run CodedUI tests on this machine, which has the test agent installed, from Visual Studio I could not, because Visual Studio cannot connect a test controller registered for test manager infrastructure, or the controller does not allows connection. I don't understand the point of it.