CODEWITHSUNDEEP
jenkinsgroovyjenkins-pipeline
rismoney
rismoney

Reputation: 531

throttling jenkins parallel in pipeline

I came across this message with the code below in JENKINS-44085.

If I already have a map of branches that contains 50 items, but I want to parallel them 5 at a time, how do I need to modify this code? My code already has a map of 50 items in a var named branches.

// put a number of items into the queue to allow that number of branches to run
for (int i=0;i<MAX_CONCURRENT;i++) {
    latch.offer("$i")
}

for (int i=0; i < 500; i++) {
    def name = "$i"
    branches[name] = {
        def thing = null
        // this will not allow proceeding until there is something in the queue.
        waitUntil {
            thing = latch.pollFirst();
            return thing != null;
        }
        try {
            echo "Hello from $name"
            sleep time: 5, unit: 'SECONDS'
            echo "Goodbye from $name"
        }
        finally {
           // put something back into the queue to allow others to proceed
            latch.offer(thing)
        }
    }
}

timestamps {
    parallel branches
}

Upvotes: 5

Views: 4144

Answers (1)

Ray Burgemeestre
Ray Burgemeestre

Reputation: 1228

This question is a bit old, but for me the problem was also relevant yesterday. In some cases your Jenkins jobs may be light on Jenkins but high on some other system, so you want to limit it for that system. In my opinion using max executors per build agent is not the right way to do that because if your Jenkins cluster scales you will have to adjust stuff.

To answer your question, you probably want to do something like this:

  • Create a branches map with numeric indexes "0", "1", etc.
  • In the try block of that code you pasted have something like: build(my_branches[name])

At least that's how I was using that same workaround before. But then someone at work pointed out a better solution. I also commented this simpler solution in the Jira ticket you refered to. It requires the Lockable Resources Plugin: https://wiki.jenkins.io/display/JENKINS/Lockable+Resources+Plugin

  • Go to: http://<your Jenkins URL>/configure and add X lockable resources with label "XYZ".
  • Use in your code as such:
def tests = [:]
for (...) {
    def test_num="$i"
    tests["$test_num"] = {
        lock(label: "XYZ", quantity: 1, variable: "LOCKED") {
            println "Locked resource: ${env.LOCKED}"
            build(job: jobName, wait: true, parameters: parameters)
        }
    }
}
parallel tests

The nice thing about this is that you can use this across different jobs. In our case different jobs have a load on XYZ so having these global locks are very handy.

Upvotes: 3

Related Questions