CODEWITHSUNDEEP
azure-devopscontinuous-integrationautomated-testsplaywrightazure-pipelines-yaml
ChunkyHippo
ChunkyHippo

Reputation: 61

Playwright test fails on Azure Devops Pipeline and pipeline hangs

I've created a simple pipeline in Azure Devops to run Playwright tests and explicitly coded one of the 2 tests to fail. When this happens during the pipeline execution, the pipeline seems to hang and I'm guessing will eventually timeout (after 30min, I cancelled). However when I execute tests that succeed, the pipeline exits successfully. Below is my yaml pipeline. Does anyone know why this might happen - have I missed something here?

[Edit]: I added the line below to my playwright.config.ts file and the task now fails like I expect. The default Playwright config opens the html report on any failures. There are a bunch of reporters you can use with Playwright, so take a look at what suits you (https://playwright.dev/docs/test-reporters).

reporter: [ ['html', { open: 'never' }] ],

trigger:
   - master
   - test
   - dev

pool:
   vmImage: macos-latest

jobs:
   - job: Build
     displayName: Build stage
     steps:
        - task: NodeTool@0
          inputs:
             versionSpec: '16.x'
          displayName: 'Install Node.js'
        - script: |
                npm ci
          displayName: 'Install dependencies'
        - script: |
            npx playwright install --with-deps
          displayName: 'Install Playwright'
        - script: |
            npm run test-unit-smoke
          displayName: 'Run unit tests'

enter image description here

Upvotes: 1

Views: 3467

Answers (3)

luke_19
luke_19

Reputation: 21

There are two ways:

  1. In playwright.config.ts, extend reporter config with open: 'never':

    reporter: [['html', { outputFolder: 'my-report', open: 'never' }]],

  2. In the yml file, use extend to run script with: env: CI:true :

    - script: |
npx playwright test
      displayName: 'Run Playwright tests'
      continueOnError: true
      env:
        CI: true

Upvotes: 2

I.sh.
I.sh.

Reputation: 1935

According to Playwright docs and Playwright developers answer on Playwright repo, you should config an env variable CI with value true or 1

        - script: |
            npm run test-unit-smoke
          displayName: 'Run unit tests'
          env:
            CI: 'true'

Please note that the answer above is preferred, because CI: 'true' effects some more configurations such as workers, forbidOnly, and retries.

Alternatively you can set the reporter property in playwright.config like so:

reporter: [ ['html', { open: 'never' }] ],

Upvotes: 2

ttqa
ttqa

Reputation: 602

The reason is that your "CI" environment variable in the virtual server is not true at the moment you are running the tests. If CI is actually true at the moment, you will never see the "Serving HTML report.." text in your Pipeline logs.

I read online that CI providers should have CI as true by default, but at least in Azure Pipelines I've always had to manually set the value to true.

I haven't tried macOS, but with Windows I have to add "set CI=true" before running the tests in the YML-file.

Example with Azure Pipelines with a Windows VM:

- task: CmdLine@2
  displayName: 'Run all tests with Chrome and Firefox'
  inputs:
    script: 'set CI=true && npx playwright test ./tests/test.spec.ts --project=chromium --project=firefox'

With macOS you could try "export CI=true" before your test.

Upvotes: 0

Related Questions