How to force Jenkins to skip a downstream job if a new build is waiting in the pipeline?

Question

My build pipeline in Jenkins is broken down into three jobs:

1. Build code
2. Deploy code to environment
3. Run automated functional tests

I have set it up so that concurrent builds can occur and the build pipeline will stop a build from entering #2, if #2 or #3 is currently running for another build.

What I want to be able to do is set up Jenkins to handle when there is more than one build waiting, and #2 and #3 finish, for only the LATEST build to enter into #2 and #3.

Is there a way to do this out of the box? IF you have the book Continuous Delivery, what I'm trying to do is implement what's on p. 118 - p. 119

Answer 1

The problem with "block upstream" or "block downstream" is that you're always blocking something which could be doing work.

If you use "git", you can do something along these lines - which happens to be what I am doing...

I use a tracking branch which points to the most recent finished build job of any step, named as follows: <branch>-latest-<step>. So, if you run a "build" step based on master, you would get master-latest-build. It is very easy to move this branch towards the end of your build script, just run: git branch -f <name> HEAD, followed by a push.

I then have the downstream jobs trigger off of that tracking branch. This way, all the jobs are loosely coupled and will do the right thing on whatever the upstream job produced, independent of what that upstream job is currently working on.

If, in addition to that, you also tag your build, your downstream job can retrieve the tag and re-use it as the build name to allow you to easily correlate various runs.

This is very effective if the pipeline has steps of vastly differing length - especially when your downstream steps take much longer than your upstream steps, which in my world is the norm, as the downstream tests include a whole suite of performance and integration tests...

Answer 2

The Workflow plugin lets you write your whole pipeline as one script. In this case the stage step can be used to control access:

stage 'build'
// any number of builds get here
stage name: 'deployAndTest', concurrency: 1
// only the last to build successfully enters here

(For visualization purposes, JENKINS-29892 would let you mark a boundary in between the deploy and test stages.)

Answer 3

Should try one of those, under Advanced Project Options:

• Block build when upstream project is building
  (should make sure it does not cause steps 2 and 3 to get stuck in queue)

• Block build when downstream project is building
  (I know this one sounds like the opposite to your request,
  but the actual result is that you accumulate changes to a single build-cycle,
  preventing extra runs)

If this causes unwanted builds to pile-up,
please review the following links that should help you
empty the queue or kill running jobs:

• Stopping Jenkins job in case newer one is started (by malenkiy_scot)

• Purge Build Queue Plugin

• Signal killer

• Kill All the Builds (courtesy of Kohsuke Kawaguchi)

  import hudson.model.*;
  
  Hudson.instance.computers.each { c ->
    c.executors.each { e ->
       e.interrupt();
    } 
  }
  

Cheers