CODEWITHSUNDEEP
jenkinslockingjenkins-pipeline
user2144067
user2144067

Reputation: 142

Jenkins Resource Locks: Get lock for multiple Node specific resources

I'd like to create multiple Resources for a certain node, or use a reusable type for several nodes.

In this case it is "RAM requirement", so the resource name e.g. would be 1GBRAM. alternatively 1GBRAM_Nodexy if I need to specify this on a per node basis.

In the end I'd like to limit the amount of concurrent Jobs based on the peak amount of memory a Job uses up on this node, to avoid hangs because of low memory on the server. And I can set the amount of RAM which is available for executors.

Different Nodes will have different amounts of RAM, and individual Jobs have different RAM requirements.

So I would like to configure each Job with its RAM requirements

lock(resource: '1GBRAM_Nodexy', quantity: 8)

Is this achievable with Pipelines and lockable resources?

Is there an alternative, better way to achieve this? Ideally, the locks can be checked before the slave is selected, and the best suited node is picked.

Read about resource locks and labels. I Did not find any Node specific section, also no possibility to acquire multiple items of the same resource.

lock(resource: '1GBRAM_Nodexy', quantity: 8)

I expect that each run of the Job locks the equivalent amount of RAM on the used slave node. If not enough "RAM" units are used up, a Job is not run on such a node.

Upvotes: 2

Views: 3888

Answers (2)

Dzmitry Sankouski
Dzmitry Sankouski

Reputation: 187

Addition to accepted answer(edit queue is full):
As for selecting specific node because it has RAM available -- i.e., your "the locks can be checked before the slave is selected, and the best suited node is picked" statement, a org.jenkins.plugins.lockableresources.LockableResourcesManager class may be used to check available memory on the nodes, and decide, which node to use, for example:

def nodeFreeGbThreshold = 2
def resourceManager = new org.jenkins.plugins.lockableresources.LockableResourcesManager()
def nodeAFreeGb = resourceManager.getFreeResourceAmount("NodeA_GB")
def nodeBFreeGb = resourceManager.getFreeResourceAmount("NodeB_GB")
def agentLabel = nodeAFreeGb < nodeFreeGbThreshold ? 'NodeA' : 'NodeB'

pipeline {
    agent { label 'agentLabel' }
    stages {
        stage('Build') {
            steps {
                // This assumes that all possible nodes have a label like this defined with their name in it
                lock(label: "${NODE_NAME}_GB", quantity: 4) {
                    // ... build steps
                }
            }
        }
    }
}

and for scripted pipelines:

def nodeFreeGbThreshold = 2
def resourceManager = new org.jenkins.plugins.lockableresources.LockableResourcesManager()
def nodeAFreeGb = resourceManager.getFreeResourceAmount("NodeA_GB")
def nodeBFreeGb = resourceManager.getFreeResourceAmount("NodeB_GB")
def agentLabel = nodeAFreeGb < nodeFreeGbThreshold ? 'NodeA' : 'NodeB'

node(agentLabel) {
    // This assumes that all possible nodes have a label like this defined with their name in it
    lock(label: "${NODE_NAME}_GB", quantity: 4) {
        // ... build steps
    }
}

Upvotes: 0

Nick Jones
Nick Jones

Reputation: 4475

I think you can't quite do what you're looking for, but perhaps you can come close.

First, what you want is to use label instead of resource. You'd define as many 1GB-representing resources (say, GB1, GB2, GB3, etc.) as you have RAM, giving them all the same label (say, GB), and then use a lock statement like this (e.g., if the job in question needed 4GB of memory):

lock(label: 'GB', quantity: 4)

This will lock 4 of the resources that have this GB label, waiting if needed until it's able to do so, and then will release them when leaving the locked scope.

The node-specific locking is the trickier part. If you were content with using a different label per node (NodeA_GB, NodeB_GB, etc.), and with "pinning" jobs to particular nodes, then the solution above would suffice, e.g.:

// Require 4GB of memory on NodeA
lock(label: 'NodeA_GB', quantity: 4)

What I'm not aware of a way to do is to have a specific node selected because it has RAM available -- i.e., your "the locks can be checked before the slave is selected, and the best suited node is picked" statement. But you could at least detect the node that was allocated by a regular agent statement, using env.NODE_NAME, then use that as part of your node-specific lock label:

agent any
stages {
  stage('Build') {
    steps {
      // This assumes that all possible nodes have a label like this defined with their name in it
      lock(label: "${NODE_NAME}_GB", quantity: 4) {
        // ... build steps  
      }
    }
  }
}

Incidentally, I'm using a label+quantity approach myself but in order to achieve lock-based throttling -- restricting the total number of concurrent builds across all branches of a multibranch pipeline job -- since the Throttle Concurrent Builds plugin went through a period of not being maintained and had some significant, open issues during that time.

Upvotes: 5

Related Questions