CODEWITHSUNDEEP
c#seleniumjenkins
Kev
Kev

Reputation: 360

How do I stop Jenkins from flagging a good test run as error?

I am running C#/Selenium tests in Jenkins via a batch file.

Example call "C:\FrontEnd\MobileRegression.bat"

And inside the bat file

c:\SpecRun\SpecRun.exe run Default.srprofile /baseFolder:C:\testBase /filter:"@MobileGalaxy" /log:specrun.log /outputfolder:output /report:MobileGalaxy.html

This works fine and in this examplw there are 10 scenarios to be run. Sometimes 1 or 2 of the scenarios fail but they are set to rertry twice more if this happens and always they pass on the 2nd or 3rd try. So at the end of the run the result is a Pass and because some of the scenarios failed on their first run Jenkins logs this as a fail. Is there anyway of changing this (for example if 90% of tests pass then mark as PASS)?

Thanks in advance for any help

Kev

Upvotes: 0

Views: 666

Answers (1)

JRichardsz
JRichardsz

Reputation: 16564

Jenkins ends with error because your batch file return an exit code different to zero, which means:

When a program finishes executing it returns an exit code to the system. The exit code (also called "exit status") is an integer from 0 to 255. The batch system reports this exit code. Always zero to success an non-zero to error.

According to the comments, I suggest to you, the following approaches:

#1 try catch inside your batch file

You need to ensuring that your batch ends with zero as exit code or success. You could use errorlevel and exit /b 0.

Here some theory and examples:

#2 try catch using jenkins pipeline script

In this approach you need to split your one batch file (with 10 scenarios) to 10 batch files (one by scenario)

After that, you could use jenkins pipeline script to invoke each batch file and using a try catch , control the error and also update a counter_error variable.

At the end of your jenkins script, you could evaluate :

if count_error > 9
  // notify a success execution to jenkins 
  // using currentBuild.result = 'SUCCESS'

Here some theory and examples:

Upvotes: 1

Related Questions