Selenium webdriver test 100% pass locally, 100% fails on VSTS server

Question

currently in my 2nd week of trying to work out why a certain test runs perfectly locally but fails when being kicked off by a VSTS agent.

Program flow:

• Select an row from a table
• Click Verify button
• CSS overlay darkens the screen, spinner in the middle
• Toaster pop up in the bottom left saying Verification starting
• About 1.5 minutes of time passes, spinner spins
• CSS overlay removed
• Toaster pop up bottom left, successful verification

I have screenshots and debug output on everystep of the way and the failure is at the "Waiting for CSS overlay to be removed" stage. It times out waiting for it, now matter how long the timeout.

Screenshot shows the CSS overlay and spinner still spinning even though the verification process completed successfully. Its almost as if it's stuck in time.

Scenarios and results:

• Run from Local Visual Studio - PASS
• VSTS Agent - FAIL
• Manually run on VSTS Agent by remoting in and passing the exact commandline as the build - PASS (This is what infuriates me!!)
• Downgrading / upgrading Webdriver - no difference
• Headless / full mode - no difference
• Using different types of waits - no difference
• Waiting for Toaster pop ups instead of CSS - no difference

Answer 1

VSTS will run Selenium and the browser as local admin. The browser will refuse that when started in sandbox mode.

Check for articles related to running Selenium as local admin

Answer 2

Please check Azure DevOps Hands-On-Labs to get started with Automating Selenium Tests in Azure Pipelines. Hope this helps...

Answer 3

I'll answer my own question: We created a new agent which was an auto login. We watched the test run and pressed F12 and there, in the console, was an error.

It was trying to trim() on the username, the test account looked like it had a username but I guess it didn't really. So the system, when run by the agent, was throwing an error and not handling it.

Handled the error and everything went through swimmingly. Very long and frustrating road to get to here! Hope this helps someone in the future.

Answer 4

One thing to note from the documentation:

Agents that are configured to run as service can run Selenium tests only with headless browsers. If you are not using a headless browser, or if you are running UI tests for desktop apps, Windows agents must be configured to run as an interactive process with auto-logon enabled.

The fact that you also tried to run Headless might mean that even if you are running as a service and still got errors, this might not be your issue. In some cases, I have had issues with remoting to the machine messing up future test runs or video recording by logging out the user or changing the screen resolution. The documentation continues about ways to prevent this:

If you use Remote Desktop to access the computer on which an agent is running with auto-logon, simply disconnecting the Remote Desktop causes the computer to be locked and any UI tests that run on this agent may fail. To avoid this, use the tscon command on the remote computer to disconnect from Remote Desktop. For example:

%windir%\System32\tscon.exe 1 /dest:console

In this example, the number '1' is the ID of the remote desktop session. This number may change between remote sessions, but can be viewed in Task Manager. Alternatively, to automate finding the current session ID, create a batch file containing the following code:

for /f "skip=1 tokens=3" %%s in ('query user %USERNAME%') do (  
    %windir%\System32\tscon.exe %%s /dest:console 
) 

Save the batch file and create a desktop shortcut to it, then change the shortcut properties to 'Run as administrator'. Running the batch file from this shortcut disconnects from the remote desktop but preserves the UI session and allows UI tests to run.