CODEWITHSUNDEEP
jenkinsjenkins-pipeline
David West
David West

Reputation: 2328

Jenkins Pipeline and Groovy. Iterate through things in parallel until they are done

I am sketching out a Jenkins pipeline. I can run tests in groups of 8 on 5 parallel nodes until they are done.

Below I'm simplifying this to do groups of 8 on 3 nodes.

I want to know if I can loop stages in a pipeline to complete them until I've gone through each.

TESTS.collate(8).collate(3).each {
  parallel {
    stage('run tests 1') {
      agent { label 'node1' }
      runTests(it[0])
    }
    stage('run tests 2') {
      agent { label 'node2' }
      runTests(it[1])
    }
    stage('run tests 3') {
      agent { label 'node3' }
      runTests(it[2])
    }
  }
}

Upvotes: 1

Views: 1116

Answers (1)

David West
David West

Reputation: 2328

This is a relevant part of my Jenkinsfile 

                def splitNBTests = NB_TESTS.split("\n").toList().collate(7)
                println(splitNBTests)
                if (!params.runCukes) {
                    echo "SKIPPING NEW BUSINESS RUN IN OZ JOB DUE TO DISABLED PARAMETER"
                } else {
                    SUITE_RUN_ID = UUID.randomUUID().toString()
                    def commands = splitNBTests.collect { def testsForNode ->
                        createCmdLineForCukes(testsForNode, PROFILE)
                    }
                    try {
                        TAGS = '"@regression ~@wip"'
                        SCENARIO_COUNT = "${DIRECTORY_CHANGE} && bundle && bundle exec rake run_scenario_count[${TAGS},${PROFILE}] SUITE_RUN_ID=${SUITE_RUN_ID}"
                        sh(returnStdout: true, script: SCENARIO_COUNT)
                        def stepsForParallel = [:]
                        commands.toList().eachWithIndex { def myCommand, int i ->
                            stepsForParallel["RunTests${i}"] = {
                                stage("RunTests${i}") {
                                    node {
                                        label 'grange-jenkins-slave'
                                        sh("mount -a")
                                        sh(myCommand)
                                    }
                                }
                            }
                        }
                        parallel stepsForParallel

I have our Jenkins Master on Kubernetes with the Kubernetes plugin to dynamically scale slave pods. For our current regression run this could bring up about 24 nodes if we run 7 tests on each but we limit it to run 10 at a time for now.

The 'white-listing' I was talking about was under In-Process Script Approval. I currently have it set to staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getAt java.lang.Object java.lang.String.

I'm guessing that should be locked down a bit more. Tips appreciated. I come to Groovy from Ruby and hacked this together with minimum time to really learn Groovy. I'll say it was similar enough and it was definitely useful to get help from the Groovy community so I could learn of the Groovy equivalent to #each_slice in Ruby, #collate. Thanks!

Upvotes: 1

Related Questions