CODEWITHSUNDEEP
jenkinsjenkins-pipeline
warbaque
warbaque

Reputation: 633

How can I limit number of certain pipelines running in parallel?

I want to allow running following pipeline in parallel, but I have to limit maximum number of parallel runs due to limited resources.

pipeline {

  agent { label "$JENKINS_AGENT" }
  parameters { .. }
  options { .. }

  stages {
    stage('Checkout') { .. }
    stage('Config') { .. }
    stage('Deploy') { .. }
    stage('Test') { .. }
  }

  post {
    failure { .. }
    success { .. }
    always {
      cleanup()
    }
  }
}

For example if I am limited to 3 deployments, how could I achieve this?

Edit: I have added currently the following to my jenkinsfile. Seems to work, but I'm not sure if that's the correct way to do this. (using Throttle Concurrent Builds Plugin)

properties([
  [
    $class: 'ThrottleJobProperty',
    categories: ['parallel_pipeline'],
    limitOneJobWithMatchingParams: false,
    maxConcurrentPerNode: 3,
    maxConcurrentTotal: 3,
    paramsToUseForLimit: '',
    throttleEnabled: true,
    throttleOption: 'project'
  ],
])

Upvotes: 6

Views: 15693

Answers (2)

Nicola Ben
Nicola Ben

Reputation: 11377

You may find useful these plugins:

  1. Throttle Concurrent Builds Plugin
  2. Lockable Resources Plugin

Upvotes: -1

jayhendren
jayhendren

Reputation: 4518

Strictly speaking, this is not possible at the moment. There is an open ticket in the Jenkins JIRA with a request for this feature.

You may be able to get around this issue with some hacks. There are a couple workarounds in the linked JIRA issue that involve creating pseudo-semaphores in order to lock individual threads until the pseudo-semaphore is free.

Another way that I've been able to handle this in specific use cases is to ensure that each branch of the parallel step is wrapped in a stage block. Since a stage locks a single executor at a time, this means you will run as many branches as you have executor slots available, and then other branches will wait until an executor slot is free to resume execution. This doesn't always work, however - for instance, you may have reasons for not making each branch an independent stage.

Upvotes: 4

Related Questions