CODEWITHSUNDEEP
jenkins
Paul Nathan
Paul Nathan

Reputation: 40299

Multiple build queues in Jenkins

I have a number of different jobs in Jenkins, partitioned out into projects with their own "pipelines" of build/test/analysis. Most of these jobs are actually remote commands instead of on-box-builds.

However, Jenkins out of the box only supports one queue for all builds. I want to define one queue per project (or view).

How would I accomplish this?

Upvotes: 9

Views: 31105

Answers (6)

L1ghtk3ira
L1ghtk3ira

Reputation: 3179

I am thinking the Build Blocker Plugin is what you are looking for.

Github Build Blocker Plugin

Jenkins Build Blocker Plugin Information

This plugin can be installed through Manage Plugins in Jenkins.

Upvotes: 0

dskrvk
dskrvk

Reputation: 1378

First of all, I'd like to point out that, like other answers describe, pegging the different build queues to different slaves would be the recommended solution, however there's a way to accomplish the same effect with just the master node.

There's a plugin called Throttle Concurrent Builds. It allows setting the build concurrency level either on a per-job basis, or across jobs.

The documentation was a bit vague, so these are the steps that worked for me:

  1. After installing the plugin, go to Manage Jenkins -> Configure System.
  2. In the Throttle Concurrent Builds section define the job groups and their respective concurrency levels.
  3. Now for each job you want to throttle, go to its configuration, enable Throttle Concurrent Builds (General section), select Throttle this project as part of one or more categories, and assign one or more of the categories defined in the previous step (no need to set the limits here again).

Upvotes: 2

Prasanna
Prasanna

Reputation: 29

I agree with @sti. Consider multiple projects, if this is still a need. Also consider https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin to cluster them to make them seamless.

Upvotes: 0

Andrew Marentis
Andrew Marentis

Reputation: 1

I know this thread is old, but I thought I would comment anyway.

IMO

I would use one Master and multiple slaves. But manage the tool configuration requirements through a configuration management tool or CMT like(Chef, Puppet, Ansible, Etc.) I would lock down the pipelines to specific build slaves via tags for specific requirements(windows, linux, Mac, visualstudio, maven, android-sdk, etc.). Since the slaves are configured in your cmt you can simply spin up a new machine very easily regardless if it is a VM or physical hardware. One Jenkins master can handle 200+ build slaves.

Upvotes: 0

sti
sti

Reputation: 11075

What you want can only be accomplished by running a separate Jenkins master for each project.

Usually people think a Jenkins master has more administrative overhead than a slave, but if that does not hold for you, you can run multiple masters on a server, just assign them different ports.

If this isn't good for you, then maybe Jenkins isn't the right tool for you. It is not the only CI server out there. Jenkins is very easy to set up but on the other hand it is not possible to do deep customization, like multiple build queues.

Upvotes: 0

gareth_bowles
gareth_bowles

Reputation: 21130

As far as I know this is not possible without changing the Jenkins code, but I think you could achieve the same goal with minimal maintenance using build slaves. Different builds can run concurrently on the slaves, or even on the same slave if you define multiple executors (if the slave machine has >1 CPU). You can label the slaves to control which jobs get executed on each one, so you could have a separate set of slaves for each of your pipelines.

Apart from the overhead of making sure the basic slave machines stay running, the Jenkins specific overhead for running a slave is minimal. You can use a process on the master to keep the slave JAR file and the build tools you need updated; at my shop we use a simple rsync script that runs every time the master or slave is restarted to copy the latest tools from the master to the slave and restart the slave process.

This approach also reduces the extent to which the Jenkins master is a single point of failure.

Upvotes: 6

Related Questions