CODEWITHSUNDEEP
buildbot
rodee
rodee

Reputation: 3161

How to run steps in parallel in Buildbot

In Buildbot, is there a way run steps within a builder in parallel? I couldn't find any documentation on that.

For example, I want these 2 steps to be executed in parallel.

factory.addStep(Parallel(
    steps=[
        ShellCommand(
            name='test.1',
            command=["run", "test1"]
        ),
        ShellCommand(
            name='test.2',
            command=["run", "test2"]
        )
    ]
))

I couldn't find parallel implementation in steps.py

Upvotes: 0

Views: 66

Answers (1)

Dew Mihisara
Dew Mihisara

Reputation: 250

It is not inherent to Buildbot to allow steps to execute simultaneously within of a single builder. The Buildbot step system is made to run the build step by step so that it has a linear flow and appropriate logging. However, you can achieve parallelism in other ways by using workarounds.

You can implement custom Buildbot phase that controls parallel execution may be created using Python's threading and asyncio. This means that in order to execute several instructions simultaneously, the run function of a step must be overridden. Here is an example:

from twisted.internet.defer import DeferredList
from buildbot.steps.shell import ShellCommand
from buildbot.process.buildstep import BuildStep

class Parallel(BuildStep):
    def __init__(self, steps, **kwargs):
        super().__init__(**kwargs)
        self.steps = steps

    def run(self):
        deferreds = []
        for step in self.steps:
            cmd = ShellCommand(**step)
            deferreds.append(self.build.addStepsAfterCurrent([cmd]))
        return DeferredList(deferreds)

Instead of running parallel steps within a single builder, create multiple builders, each responsible for a single task. You can then orchestrate them using a parent builder or Buildbot's scheduler to trigger them concurrently.

Upvotes: 0

Related Questions