CODEWITHSUNDEEP
jenkinsjenkins-declarative-pipeline
DiogoLG
DiogoLG

Reputation: 257

How to use an executor for each stage inside a declarative parallel step

I'm trying to adapt some old multi-job in a declarative Pipeline. I would like that every stage on parallel step in a Declarative Pipeline use its own executor.

The behaviour that I'm getting is that Jenkins is using the same executor to run all the stages.

pipeline {

agent { label 'macmini'}
    stages() { 
        stage('SCM') {
            steps{
                //clone repo
            }
        }
        stage('Heavy build Stage'){

            parallel {

            stage ("First Job"){ 
                steps {  
                    //do some heavy work
                    sleep 100
                }
              }
              stage ("Second Job"){
                  steps {
                  //do some heavy work
                        sleep 100
                      }
                  }
            }
        }
    }
}

In the above example, I want that stage ("First Job") and ("Second Job") use it's own executor. But Jenkins is using only one executor to run both stages... it's also important that they use the same workspace.

If I set an agent/node after each stage, they will run in its own executor but it will also create a new Workspace (my-job@2).

Edit

Just to make myself clear, I'm used to work with executor to balance the work of a job between my different slaves. I know (empirically) that each slave can run 4 stages in parallel. Those jobs are trigger by a webhook.

So if a job starts I will get

slave 1
1. idle
2. idle
3. (Heavy Build Stage) #1
4. idle

If three more jobs are trigged I will get the following scenario

slave 1
1. (Heavy Build Stage) #2
2. (Heavy Build Stage) #3
3. (Heavy Build Stage) #1
4. (Heavy Build Stage) #4

Which means that I'm with 8 stages running (2 stages in parallel * 4 builds)

What I'm looking for is a way to split every stage to use it's own executor, so I can get the following scenario

Build Queue
1. job #3
2. job #4

slave 1
1. (Heavy Build Stage / Second Jo) #2
2. (Heavy Build Stage / First job) #2
3. (Heavy Build Stage / Second Job) #1
4. (Heavy Build Stage / First job) #1

When I try the solution of setting an agent/node after each stage

stage("First Job"){
agent{ label SLAVE_NAME }
...
}

It works but each stage runs in it's own workspace (creates a new folder with a @2), but I would like them to run in the same workspace.

I have the same behaviour when using customWorkspace option for each stage

agent {
  node {
  label "macmini"
  customWorkspace 'teste'
 }
}

The logs shows

Running in /Users/mobile/jenkins/teste
[Pipeline] {
[Pipeline] ws
Running in /Users/mobile/jenkins/teste@2

Upvotes: 2

Views: 2191

Answers (1)

Michael Kemmerzell
Michael Kemmerzell

Reputation: 5276

First you need 2 agents. Every agent get its own label ('jenkinsagent1' and 'jenkinsagent2' in the example below). The global pipeline is not given an agent. Every stage gets its own agent.

pipeline{

    agent none //this tells the pipeline that it is not executed by 1 agent

        stage('Is parallel executed on 2 agents') {
             parallel {

                 stage('foo'){
                       agent{
                          node { label'jenkinsagent1' //executed by this agent
                           customWorkspace 'myWorkspace/test' //use this workspace
                          } 
                       }
                       steps { // do something}
                  }

                  stage('bar'){
                       agent{
                          node { label 'jenkinsagent1' //execute on this agent
                        customWorkspace 'myWorkspace/test'} } //use this workspace
                       steps { // do something different} //executed by the other agent
                  } //closing stage bar
    } //closing parallel
// closing pipeline

Be careful if you want to add a 3rd or 4th stage. You always have to set the agent for every new stage. If you don't do it the pipeline won't work.

Upvotes: 1

Related Questions