How to run dynamic stages in paralell on jenkins with a separate kubernetes agent for each stage

Question

I tried combining things I have found on the syntax but this is as close as I can get. It creates multiple stages but says they have no steps.

I can get it to run a bunch of parallel steps on the same agent if I move the agent syntax down to where the "test" stage is defined but I want to spin up separate pods for each one so I can actually use the kubernetes cluster effectively and do my work parallel.

attached is an example Jenkinsfile for reference

def parallelStagesMap

def generateStage(job) {
    return {
        stage ("$job.key") {
            agent {
                kubernetes {
                    cloud 'kubernetes'
                    yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: name
    image: image
    command:
    - sleep
    args:
    - infinity
"""
                }
            }
            steps {
                sh """
                    do some important stuff
                """
            }
        }
    }
}

pipeline {
    agent none
    stages {
        stage('Create List of Stages to run in Parallel') {
            steps {
                script {
                    def map = [
                        "name" : "aparam",
                        "name2" : "aparam2"
                    ]
                    parallelStagesMap = map.collectEntries {
                        ["${it.key}" : generateStage(it)]
                    }
                }
            }
        }
        stage('Test') {
            steps {
                script {
                    parallel parallelStagesMap 
                }
            }
        }
        stage('Release') {
            agent etc
            steps {
                etc
            }
        }
    }
}

Answer 1

To run your dynamically created jobs in parallel you will have to use scripted pipeline syntax.
The equivalent syntax for the declarative kubernetes agent in the scripted pipeline is podTemplate and node (see full Doucumentation):

podTemplate(yaml: '''
    apiVersion: v1
    kind: Pod
    spec:
      containers:
      - name: maven
        image: maven:3.8.1-jdk-8
        command:
        - sleep
        args:
        - 99d
''') {
  node(POD_LABEL) {
     ...
    }
}

Notice that the podTemplate can receive the cloud parameter in addition to the yaml but it defaults to kubernetes so there is no need to pass it.

So in your case you can use this syntax to run the jobs in parallel on different agents:

// Assuming yaml is same for all nodes - if not it can be passed as parameter
podYaml= """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: name
    image: image
    command:
    - sleep
    args:
    - infinity
"""

pipeline {
    agent none
    stages {
        stage('Create List of Stages to run in Parallel') {
            steps {
                script {
                    def map = ["name" : "aparam",
                              "name2" : "aparam2"]
                    parallel map.collectEntries {
                        ["${it.key}" : generateStage(it)]
                    }
                }
            }
        }
    }
}

def generateStage(job) {
    return {
        stage(job.key) {
            podTemplate(yaml:podYaml)  {
                node(POD_LABEL) {
                    // Each execution runs on its own node (pod)
                    sh "do some important stuff with ${job.value}"
                }
            }
        }
    }
}

Answer 2

As explained in this answer:

Dynamic parallel stages could be created only by using Scripted Pipelines. The API built-it Declarative Pipeline is not available (like agent).

So, you can't run dynamic stages in parallel on different agents.

To achieve what you want to do, a solution would be to trigger another pipeline that run on a new kube pod and wait for its completion before next steps.

Here is the Jenkinsfiles for more understanding:

• Main job Jenkinsfile:

def parallelJobsMap

def triggerJob(item) {
     return {
        build job: 'myChildJob', parameters: [string(name: 'MY_PARAM', value: "${item.value}")], wait: true
     }
}

pipeline {
    agent none
    stages {
        stage('Create List of Stages to run in Parallel') {
            steps {
               script {
                    def map = [
                        "name" : "aparam",
                        "name2" : "aparam2"
                     ]
                     parallelJobsMap = map.collectEntries {
                         ["${it.key}" : triggerJob(it)]
                     }
                }
            }
        }
        stage('Test') {
            steps {
                script {
                    parallel parallelJobsMap
                }
            }
        }
        stage('Release') {
            agent any
            steps {
                 echo "Release stuff"
            }
        }
    }
}

• Child job Jenkinsfile:

pipeline {
    agent none
    parameters {
      string(
        name: 'MY_PARAM',
        description: 'My beautiful parameter',
        defaultValue: 'A default value',
        trim: true
      )
    }
    stages {
        stage ("Job") {
            agent {
                kubernetes {
                    cloud 'kubernetes'
                    yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: name
    image: image
    command:
    - sleep
    args:
    - infinity
"""
                }
            }
            steps {
                echo "Do some important stuff with the parameter " + params.MY_PARAM
            }
        }
    }
}