CODEWITHSUNDEEP
jenkinsjenkins-pipelinejenkins-groovy
Good Night Nerd Pride
Good Night Nerd Pride

Reputation: 8490

Jenkins pipeline: execute steps in parallel dynamically

In our Jenkins pipeline we use the following function to execute a set of deployment tasks in parallelized chunks:

def runParallel(tasks, count) {
    while (tasks.size() > 0) {
        parallel tasks.take(count)
        tasks = tasks.drop(count)
    }
}

With the count parameter we control how many of the tasks are executed in parallel, so the network isn't overwhelmed with too many parallel deployments.

This works nicely, but has the downside that if one task in a chunk takes a very long time, the next chunk has to wait until the previous chunk is completely finished. Even though just one task is currently being executed and there are enough resources available to start count - 1 new tasks.

So what we actually need is not splitting the tasks into fixed-size chunks with count tasks, but rather a sliding window approach that iterates over tasks with a window size of count and moves to the next task as soon as the first task inside the window is finished. This way we wouldn't have the wait times our chunk-based approach has.

AFAIK it's not possible to add more tasks to a parallel step after it has been started. But maybe there is a work-around?

Ideally parallel would take a parameter that controls the degree of parallelization. Then we could just do:

parallel tasks, count

Upvotes: 2

Views: 1293

Answers (1)

crash
crash

Reputation: 670

It seems you’re very lucky, since there was a plugin released some some weeks ago, with which you can archive this: Jenkins Concurrent Step Plugin.

The original issue for this problem

Example pipeline with a semaphore:

pipeline {
  agent any

  stages {
    stage('test') {
      steps {
        script {
          def semaphore = createSemaphore permit: 1
          boolean out = false
          parallel(
            semaphore1: {
              acquireSemaphore (semaphore) {
                echo "out=${out}"
              }
            },
            semaphore2: {
              acquireSemaphore (semaphore) {
                sleep 30
                echo "out=${out}"
              }
            },
            semaphore3: {
              acquireSemaphore (semaphore) {
                sleep 10
                echo "out=${out}"
              }
            },
            semaphore4: {
              acquireSemaphore (semaphore) {
                sleep 10
                echo "out=${out}"
              }
            }
          )
        }
      }
    }
  }
}

Upvotes: 1

Related Questions