How to check status of specific stage (of previous build) during Jenkins build?

Question

In Jenkins (multi-branch pipeline), I want to be able to find the most recent build in which some stage was successful (e.g. not skipped, not failed). I can easily loop through the previous build and check if the build itself was successful, but I do not know how to check if a given stage was successful for that build.

Collection<String> getChangedFilesSinceLastSuccessfulStageBuild(String... stages) {
    def files = new HashSet<String>()
    def build = currentBuild

    // look at previous build(s) until we find a successful build AND the given stage
    // (or leafiest child stage) was also successful (or we reach the beginning of time)
    while (build != null && build.result != 'SUCCESS' /* ... unknown here ... */) {
        // ... round up the changed files, etc. (easy)
    }

    // ... more easy things here

    return files;
}

This would be called as part of the process to ask the question:

Have any API-related files changed since the last successful API build?

It would be called as (e.g.):

hasApiChanges = getChangedFilesSinceLastSuccessfulStageBuild("Build/Test API")
hasUiChanges = getChangedFilesSinceLastSuccessfulStageBuild("Build/Test API")

(Caveat -- the "Build/Test API" stage is actually a "Build/Test" stage which has parallel stages within it for "API" and "UI". So I really need to find the "API" stage within the "Build/Test" stage.)

Not sure how to find a (nested) stage by name and then check its result.

Alternatively, during that stage I could write some persistent build metadata to the current build, and then check if that metadata exists / is true during subsequent builds. (Not sure how to do this, either.)

PS: I can't seem to find the relevant Javadoc for this (multi branch pipeline) -- would love a link to that.

Answer 1

I ended up implementing this feature without tracking success at the step level. I essentially look back through the recent builds and for any successful build, if it has changes to the "affected" files and before that build there were also changes to the affected files, then the build continues for that set of files (e.g. API).

I also added parameters to the pipeline so that we can override this rule as needed.

Set Up Variables

final String forceApiBuildParam = 'forceApiBuild'
final String skipApiBuildParam = 'skipApiBuild'
final String forceUiBuildParam = 'forceUiBuild'
final String skipUiBuildParam = 'skipUiBuild'
final String apiPathRegex = /^(api\/|build.gradle|checkstyle.xml|settings.gradle|Jenkinsfile).*/
final String uiPathRegex = /^(ui\/|Jenkinsfile).*/
Boolean apiBuildEnabled = params[forceApiBuildParam] == true
    ? true
    : (params[skipApiBuildParam] == true 
        ? false 
        : hasChangesInPathsSinceLastSuccessFulBuildWithChangesInPaths(verbose, apiPathRegex))
Boolean uiBuildEnabled = params[forceUiBuildParam] == true
    ? true
    : (params[skipUiBuildParam] == true 
        ? false 
        : hasChangesInPathsSinceLastSuccessFulBuildWithChangesInPaths(verbose, uiPathRegex))

Set Up Parameters

parameters {
    booleanParam(name: forceApiBuildParam, defaultValue: false, description: 'Whether or not to force the API to build and deploy.')
    booleanParam(name: skipApiBuildParam, defaultValue: false, description: 'Whether or not to skip the API build, even if there are changes.')
    booleanParam(name: forceUiBuildParam, defaultValue: false, description: 'Whether or not to force the UI to build and deploy.')
    booleanParam(name: skipUiBuildParam, defaultValue: false, description: 'Whether or not to skip the UI build, even if there are changes.')
}

Function to Check for Changes (can likely be simplified)

/**
 * Attempts to determine if, for a set of paths given in the form of a regex, 
 * any files have changed (which match those paths) since the last build which
 *  was successful and also had changes within those paths.
 *
 * @param verbose true to include additional logging
 * @param pathRegex the regex of the paths to consider
 * @return true if paths matching the provided regex have changed since the last
 *              successful build which also had changes to a file matching the
 *              regex; false if otherwise or if no successful build yet exist
 */
boolean hasChangesInPathsSinceLastSuccessFulBuildWithChangesInPaths(boolean verbose, String pathRegex) {
    Collection<String> allChangedPaths = new HashSet<String>()
    RunWrapper build = (RunWrapper) (Object) currentBuild
    int buildCount = 0

    echo "Checking for changed paths since last successful build for paths matching: ${pathRegex}"

    while (build != null) {
        if (verbose) echo "Checking build ${build.displayName}..."

        WorkflowRun rawBuild = (WorkflowRun) build.rawBuild
        String buildHash = rawBuild.getAction(BuildData.class).lastBuiltRevision.sha1String.substring(8)
        boolean success = build.result == 'SUCCESS'

        if (success) {
            echo "Found successful build (${build.displayName}; #${buildHash}) -- ${buildCount} builds ago."
        }
        else if (verbose) {
            echo "Found failed build (${build.displayName}; #${buildHash}) -- ${buildCount} builds ago."
        }

        for (GitChangeSetList changeLog in (build.changeSets as List<GitChangeSetList>)) {
            Collection<String> changedChangeLogPaths = new HashSet<String>()

            for (GitChangeSet entry in changeLog) {
                if (verbose) echo "Checking commit ${entry.commitId}..."

                // if this build is successful, check if the already-stored files match `pathRegex`
                // if so, then there have been changes since this successful build
                if (success) {
                    for (String thisBuildPath in entry.affectedPaths) {
                        if (verbose) echo "Checking path ${thisBuildPath} for commit ${entry.commitId}..."

                        // check if this file (changed in this build) matches
                        if (thisBuildPath.matches(pathRegex)) {
                            if (verbose) echo "Match found!"

                            // now look through all changed files *before* this build for matches
                            for (String allBuildsPath in allChangedPaths) {
                                if (verbose) echo "Checking changed path ${allBuildsPath}..."

                                // check if this file (which was changed before this build) matches
                                if (allBuildsPath.matches(pathRegex)) {
                                    echo "Since build ${build.displayName}, path \"${allBuildsPath}\" was changed and matches: ${pathRegex}"

                                    return true
                                }
                            }

                            // if we get here, then we found a successful build matching `pathRegex`, but since then
                            // there have been no files changed which match `pathRegex` -- so no need to build again
                            echo "No files changed since build ${build.displayName} matching: ${pathRegex}"

                            return false
                        }
                    }
                }

                // add the affected paths to the list of changes for the changeset
                changedChangeLogPaths.addAll(entry.affectedPaths)
            }

            // if we get here then the build was not successful or had no files changed whose
            // path matches `pathRegex` -- add the files to the list and continue
            allChangedPaths.addAll(changedChangeLogPaths)
        }

        build = build.previousBuild
        buildCount++
    }

    echo "Found no successful builds (checked ${buildCount} builds) with changes to paths matching: ${pathRegex}"

    // return true to signify so we can do the initial build
    return true;
}

There is one caveat with this approach, and it is that it may build slightly more often than it needs to. Say there is an API change which breaks the build -- not because of an issue in the API code change, but because of an issue in the build script, or deployment of the API (something external to the code change). Then the API will be rebuilt on each subsequent build until another API change comes in.

• Build #100 -- API change, broken build
• Build #101...#n -- no API change; API rebuilt due to #100 failure -- now successful
• Build #102...#n -- no API change; API rebuilt due to #100 failure -- even though #101 was successful since then
• Build #n+1 -- API change, rebuild API
• Build #n+2 -- no API change; API not rebuilt because #n+1 has API change and was successful

In the example above, the API is rebuilt starting from #101 until the next API change comes in and results in a successful build. I don't think this is possible to fix with this solution unless it is possible to tag, for every build, whether or not the API/UI built successfully. If we could do that, then #102+ would not have been built because we know that the API build was successful on #101, even though no API files were affected during that build.