CODEWITHSUNDEEP
azure-devopsalluretest-reporting
QA_hacks
QA_hacks

Reputation: 297

How to implement Allure report in Azure Devops?

I am running some API tests which generates a customized xml junit report in the Azure devOps pipelines. Junit report is visible under the 'Tests' tab. But there I see the tab "Allure report" and I want to implement that in my framework. How do I do it? I want to use my current Junit report to somehow visible in the "Allure Report" section so that I can make the report more interactive. Attached is the page in Azure devops I am talking about.

Azure devops Pipeline stats

Upvotes: 3

Views: 19252

Answers (2)

Alienzzz
Alienzzz

Reputation: 27

Well, it depends on where the Test is coming from. If you are getting the test from a .json file post-man collection, you need to look into this package here for more explanation about my steps.

If you are to run the test using the playwright method, then look into this material here for more details.

For the Azure-DevOps Allure extension, the above-mentioned by @Hugh Lin (Allure Test Reports) personally, I consider it entirely useless on Azure DevOps, maybe as an integration with Visual Studio or VSCode IDEs it works, but for Azure DevOps? it is entirely worthless. Even the reviews on it would say as much, so instead, I recommend Allure Report Viewer by Micheal Clay.

I will only talk about report generation coming from Newman Postman collections, if you are not using this method, I don't think anything would be helpful from this point below.

So Steps

  • Create an Azure Repository
  • Go to your Newman Postman collection and download both the collection.json and the environment.json file, then upload it to the above-created repository
  • Create a new pipeline and link it to the repository (Choose Classic Editor). Note: I will only be giving scripts that were added by me, Display names of Agent tasks are at your discretion.
  • Add a command line Task. script: "npm install -g newman@latest".
  • Add NPM task; Command: custom; Command and Arguments: "install -g newman-reporter-allure@latest allure-commandline@latest". Note: allure-command line package is needed to run the "allure generate" command down the line.
  • Add a command line Task. script: "newman run https://api.postman.com/collections/<collectionID>?access_key=PMAT-<Token> --environment environment.json -r allure --reporters cli,junit,json,allure --reporter-junit-export Results\junitReport.xml". Note: I did not use the collection.json, I am running the tests directly from the QA personnel's postman collection, so that if there are any updates or changes on the collection it reflects on the pipeline and I don't have to keep manually uploading collection.json files(What can I say, I am a lazy DevOps Engineer. 'Lol, whilst scratching my head'). Also please remember to enable "continue on error" by checking it under Control Options, so that the pipeline keeps running even if there are failed tests. Run test
  • Add a command line Task. script: "allure generate --clean".
  • Add a publish allure report Task. Allure file Directory: "$(System.DefaultWorkingDirectory)/allure-report/". Publish Allure Report
  • Add a publish test result Task(This is for the JUnit). Test Result files: "$(System.DefaultWorkingDirectory)\Results\*.xml". Publish JUnit Report
  • OPTIONAL Add a publish pipeline artifact task. Note: This is only useful if you want to use files created during the run on the release pipeline. I used it to send reports via SendGrid. (This should be another topic entirely.)

Results

DrawBacks

  • I do not post so often on here, hence my inability to post pictures and more than 8 links here as I need 10 reputations to do that. I posted the pictures on Cloudinary to provide public links to the pictures and made links to them. Hope it works. I will provide Links to the necessary pictures
  • I noticed that Allure Reports under my implementations don't show the total number of assertions, it just shows the number of test cases unlike that of the JUnit. I am not a QA personnel and my knowledge in that department is not crispy. Please refer to a QA personnel to look into it, that is if it is an issue. Good luck and Cheers

Upvotes: 2

Hugh Lin
Hugh Lin

Reputation: 19471

How to implement Allure report in Azure Devops?

You need to install Allure Test Reports extension first.

Allure is based on standart xUnit results output but adds some supplementary data. Any report is generated in two steps. During test execution (first step) a small library called adapter attached to testing framework saves information about executed tests to XML files. Allure already provide adapters for popular Java, PHP, Ruby, Python, Scala and C# test frameworks.

During report generation (second step) XML files are transformed to HTML report. This can be done with command line tool, plugin for CI or build tool.

Here are examples and documentation for more details.

Upvotes: 3

Related Questions