CODEWITHSUNDEEP
jenkinshpcslurm
rgov
rgov

Reputation: 4359

Jenkins on-demand build agents with SLURM

I have access to an HPC cluster that uses SLURM to schedule jobs. I would like to evaluate having Jenkins submit jobs to the cluster.

As a simple test, it works to add the HPC login node to Jenkins, and configure a Jenkins job that executes srun ... ./myprogram on the login node. The srun command tells SLURM to assign a compute node and execute ./myprogram on it.

This is not an ideal solution. Every Jenkins job has to be written to use srun. Scripts of more than one command are difficult. The compute node is not running the Jenkins agent so features are limited.

Can I have Jenkins instead spawn new nodes on-demand whenever a job is in the queue? It could then execute srun ... jenkins-agent on the login node to spawn a Jenkins agent on some compute node. And then the compute node would connect back to Jenkins and take the enqueued job. I would briefly a node hpc-job-1643 appear and then it would disappear after the job completes.

Upvotes: 3

Views: 2061

Answers (1)

Steve Byan
Steve Byan

Reputation: 41

Yes, you can have Jenkins launch nodes on-demand through Slurm.

  1. Create a new node. Set the remote root directory to something accessible from all nodes of your Slurm cluster.

  2. Set a label that your jobs will use to bring up this node.

  3. Set Usage to "Only build jobs with label expression matching this node".

  4. Select the "Launch agents via SSH" launch method (you need a plugin for this, I think)

  5. Set the host to the hostname of your Slurm control node.

  6. Set up the credentials and host key verification strategy as appropriate.

  7. Click the "Advanced ..." button for the SSH launch method.

  8. Set the Prefix Start Agent Command to srun --pty [options for whatever resources and constraints you need] sh -c '. Note the single quote at the end.

  9. Set the Suffix Start Agent Command to '. That's a single quote.

  10. For Availability, select "Bring this agent online when in demand, and take offline when idle". You'll need to set an in-demand delay (I use 0) and an idle delay.

Upvotes: 4

Related Questions