Playwright test results not publishing in Azure pipelines

Question

We have number of playwright tests that execute as part of our deployment pipeline. The yml file that kicks off the tests also has the following tasks:

  - task: PublishBuildArtifacts@1
    displayName: "Playwright-Publish Traces"
    condition: or(failed(), canceled())
    inputs:
      PathToPublish: "${{ params.testDir }}/test-results"
      ArtifactName: "Playwright Test Traces"
      publishLocation: "Container"

  - task: PublishTestResults@2
    displayName: "Playwright-Publish Results"
    inputs:
      testResultsFormat: 'JUnit'
      testResultsFiles: '*.xml'
      searchFolder: "${{ params.testDir }}/test-results"
      testRunTitle: "Playwright Tests ${{ params.shardIndex }}/${{ params.totalShards }}"  
    condition: always()

After the test run completes, I can see that the test traces are published in the artifacts section of Azure, but not the test result report (screenshots below). Why is that?

If I run the playwright tests locally, then I can see a results.xml published in the test-results directory as expected.

Looking at the logs of the Playwright-Publish Results step in the pipeline looks as though everything went well:

I did notice that if I follow the url linked in the pipeline logs, I get a 404 error though, so perhaps they're not publishing to the right place? Looking at the docs for this task, I can't see an option to specify a publishLocation like you can for the PublishBuildArtifacts task though.

Edit Just to clarify, I can see the test results presented in Azure under the tests tab, but I want to retrieve a copy of the raw results.xml.

Answer 1

After the test run completes, I can see that the test traces are published in the artifacts section of Azure, but not the test result report (screenshots below). Why is that?

As per the artifact name Playwright Test Traces, it comes from your PublishBuildArtifacts@1 task in your yaml. The task will create the artifact not publish the test result, the task works as expected.

For last screenshot of task PublishTestResults@2, it will publish the test as you wanted, but it has the message: No build level attachments to publish.

But on your summmary page, it actually has tests rate:

1. Please double confirm on Tests tab, to check if there's any test result:

2. You have published the folder content in the artifact Playwright Test Traces, it's recommended to check its content to double confirm test result xml is correctly generated or contained. Check if there are multiple xml result files. If you have, you can specify one xml to publish in PublishTestResults@2 task, or use one more PublishTestResults@2 task to publish the other xml result.

Edit:

I created the Playwright test sample project via command npm init playwright@latest. As per the doc， added the output format in playwright.config.ts.

I have yaml pipeline below:

name: Playwright Tests
variables:
  CI: true
trigger:
  - main
jobs:
  - job: test
    timeoutInMinutes: 60
    pool:
      vmImage: "ubuntu-latest"
    steps:
      - task: NodeTool@0
        inputs:
          versionSpec: "16.x"
        displayName: "Install Node.js"
      - script: |
         npm ci
         npm install -g junit-viewer
        displayName: "Install dependencies"
      - script: npx playwright install --with-deps
        displayName: "Install Playwright Browsers"
      - script: npx playwright test
        displayName: "Run Playwright tests"

      # - script: npx junit-viewer --results=$(System.DefaultWorkingDirectory)/test-results/test-results.xml --output=$(System.DefaultWorkingDirectory)/playwright-report/index.html
      #   displayName: "Generate result.xml"

      - powershell: Get-ChildItem -Path . -File -Recurse
      - publish: $(System.DefaultWorkingDirectory)/test-results
        artifact: playwright-report
        condition: always()

      - task: PublishTestResults@2
        displayName: "Playwright-Publish Results"
        inputs:
          testResultsFormat: 'JUnit'
          testResultsFiles: '*.xml'
          searchFolder: "$(System.DefaultWorkingDirectory)/test-results"
          testRunTitle: "Playwright Tests"  
        condition: always()

The pipeline succeeds and the test run display fine:

The results.xml is correctly published to artifact.