WinAppDriver in Azure devops pipeline to test applications

Question

In the company I am working I was given the Task to design a system to automatically test Windows Applications.

My approach:

• Testproject written in c# and using the WindowsDriver from Appium
• Using SpecFlow to write Feature files (test cases)
• An AzureDevOps Pipeline to run the Tests
• Tests are run on an self hosted agent
• self hosted agent runs in a VM (Windows Server 2019 standard), the VM has admin rights to be able to install the Application it is supposed to test. (Installation via another AzureDevOps Pipeline)

The Problem:

• The WinAppDriver needs an open RDP connection and the VM can not be locked in order for the WinAppDriver to be able to click Buttons.
• an open RDP connection is not a problem, because there are workarounds to disconnect from the VM but keep the connection open.
• cannot be locked is a problem though, because the VM has admin rights it needs to be locked after 15 minutes of inactivity because of security reasons.

My thoughts:

• One possible solution which comes to my mind is to have an agent running on a physical PC which can be left unlocked forever. Then before the Testpipline starts this agent establishes a connection => the test agent is unlocked. But it seems to me that there could be much better solutions.

Any suggestions how to improve the system / make it work without anyone having to connect to the VM manually?

Answer 1

I believe you can use WinAppDriver CI task on your pipeline https://github.com/Microsoft/WinAppDriver/wiki/WinAppDriver-in-CI-with-Azure-Pipelines

In YML it should look like this:

  - task: Windows Application Driver@0
    inputs:
      OperationType: 'Start'

  #Run your tests here

  - task: Windows Application Driver@0
    inputs:
      OperationType: 'Stop'

In the page linked above there is a link on how to make your custom agent meet the requirements. If the hosted agents do not meet your requirement, try using a private agent. More information on this below.

Answer 2

Let me describe how we run our tests:

• 1st stage - run VM using webhook to Azure Portal
• 2nd stage - checkout code, start WinAppDriver console, build code, run tests, stop WinAppDriver, cleanup of env
• 3rd stage - stop VM using webhook to Azure Portal

1 and 3 are ensuring we spent only minimum amount of $$$ for best possible VMs to run tests as quickly as possible 2 is establishing environment, testing and cleaning up

By this approach we are sure that every single time everything is fresh (version of application for test and version of tests) + every component is opened again. There is no dependencies to old runs (since VM was restarted) and I can say it is working really stable and entire configuration did not take more than 5 minutes.

This may also work for You since VM will be running only for a time of Test Execution and then will be closed. This approach needs to have Azure DevOps agent installed on VM which will be running as a console (not as a service) + autologon must be on place. Without above it will be impossible to achieve proper Pipeline without human interaction.