Running Postman in Azure DevOps

Question

I'm running into a weird issue when running Newman on Azure DevOps Pipeline. Here's a summary of what's happening:

• Postman tests run fine locally
• Pipeline tests fail only on the first test

    Post
    Test A
    POST XXXXX [500 Internal Server Error, 442B, 8.6s]
    1⠄ JSONError in test-script
    Test A Copy
    POST XXX [200 OK, 692B, 8.9s]
    √  Is Successful
    √  Status Code
    √  Status Message
    ---
     #  failure    detail                      
                                               
    1.  JSONError                              
                   No data, empty input at 1:1 
                                               
                   ^                           
                   at test-script   

           

It doesn't seem to matter what the exact test is, it always fails if it's the first. As a way to demonstrate this I've copied the test that was failing so that now I had

• Test A
• Test A Copy
• Test B
• Test ...

And suddenly Test A Copy works. So it's not the contents of the test but rather the first test to be tested. All of these tests are POST's

Test A Contents:

    var jsonData = pm.response.json();
    
    pm.test("Is Successful", function() {
        pm.expect(jsonData.IsSuccessful).to.be.true;
    })
    
    pm.test("Status Code", function() {
        pm.response.to.have.status(200);
    })
    
    pm.test("Status Message", function() {
        pm.expect(jsonData.StatusMessage).eql("Document insert successful.");
    })

Nothing too fancy, so why would this fail on the first run (TEST A) but not the second (TEST A Copy). It doesn't matter which test it is, if I were to run TEST B first this would be the one to fail.

It almost looks like the first request is what's waking up the server and then everything is okay.

Answer 1

I run the Azure Devops Rest API in Postman and use the export json file to run the Postman test in Pipeline

Here are my steps to run newman in azure pipeline, you can refer to them.

Step1: Export the Collection in PostMan.

Step2: Upload the Json file(e.g. APITEST.postman_collection.json) to Azure Repo.

Step3: Create a pipeline and add the install Newman step, run Postman test step.

Example:

steps:
- script: |
   npm install -g newman@5.1.2
   
  workingDirectory: '$(System.DefaultWorkingDirectory)'
  displayName: 'Command Line Script'

- script: 'newman run TEST.postman_collection.json --reporters cli,junit --reporter-junit-export Results\junitReport.xml '
  workingDirectory: '$(build.sourcesdirectory)'
  displayName: 'Command Line Script

or run with the Newman the cli Companion for Postman task(This is an extension task).

steps:
- script: |
   npm install -g newman@4.6.1
   
  workingDirectory: '$(System.DefaultWorkingDirectory)'
  displayName: 'Command Line Script'


- task: NewmanPostman@4
  displayName: 'Newman - Postman'
  inputs:
    collectionFileSource: 'TEST.postman_collection.json'
    environmentSourceType: none
    ignoreRedirect: false
    bail: false
    sslInsecure: false
    htmlExtraDarkTheme: false
    htmlExtraLogs: false
    htmlExtraTestPaging: false

Step4: Run the pipeline and it can display the same api results as in postman.