CODEWITHSUNDEEP
jenkinspytest
Marcus Ahlberg
Marcus Ahlberg

Reputation: 1389

How to not mark Jenkins job as FAILURE when pytest tests fail

I have a Jenkins setup with a pipeline that uses pytest to run some test suites. Sometimes a test fails and sometimes the test environment crashes (random HTTP timeout, external lib error, etc.). The job parses the XML test result but the build is marked as FAILURE as long as pytest returns non-zero.

I want Jenkins to get exit code zero from pytest even if there are failed tests but I also want other errors to be marked as failures. Are there any pytest options that can fix this? I found pytest-custom_exit_code but it can only suppress the empty test suite error. Maybe some Jenkins option or bash snippet?

A simplified version of my groovy pipeline:

pipeline {
    stages {
        stage ('Building application') {
            steps {
                sh "./build.sh"
            }
        }
        stage ('Testing application') {
            steps {
                print('Running pytest')
                sh "cd tests && python -m pytest"
            }
            post {
                always {
                    archiveArtifacts artifacts: 'tests/output/'
                    junit 'tests/output/report.xml'
                }
            }
        }
    }
}

I have tried to catch exit code 1 (meaning some tests failed) but Jenkins still received exit code 1 and marks the build as FAILURE:

sh "cd tests && (python -m pytest; rc=\$?; if [ \$rc -eq 1 ]; then exit 0; else exit \$rc; fi)"

Upvotes: 2

Views: 6984

Answers (2)

Fabian Ritzmann
Fabian Ritzmann

Reputation: 1725

Your attempt does not work because Jenkins runs the shell with the errexit (-e) option enabled and that causes the shell to exit right after the pytest command before it reaches your if statement. There is a one-liner however that will work because it is executed as one statement: https://stackoverflow.com/a/31114992/1070890

So your build step would look like this:

sh 'cd tests && python -m pytest || [[ $? -eq 1 ]]'

Upvotes: 4

Marcus Ahlberg
Marcus Ahlberg

Reputation: 1389

My solution was to implement the support myself in pytest-custom-exit-code and create a pull request.

From version 0.3.0 of the plugin I can use pytest --suppress-tests-failed-exit-code to get the desired behavior.

Upvotes: 2

Related Questions