CODEWITHSUNDEEP
gitlab-ci
anol
anol

Reputation: 8953

CI jobs spanning multiple stages

How can I create a CI job that spans more than one stage, to improve parallelism?

As in the following diagram:

Diagram showing one process spanning multiple stages

The idea is that slow_build should start as early as build, but test doesn't depend on it, so test should be able to start as soon as build is done.

(Note that this is a simplification: each stage has multiple processes running in parallel, otherwise I could just bundle build and test together.)

Upvotes: 5

Views: 2092

Answers (1)

Adam Marshall
Adam Marshall

Reputation: 7649

This is now possible as of Gitlab version 12.2. By adding the keyword needs to jobs that depend on other jobs, stages can now run concurrently. The full documentation for the needs keyword is here, but an example from the docs follows: https://docs.gitlab.com/ee/ci/yaml/#needs

linux:build:
  stage: build

mac:build:
  stage: build

lint:
  stage: test
  needs: []

linux:rspec:
  stage: test
  needs: ["linux:build"]

linux:rubocop:
  stage: test
  needs: ["linux:build"]

mac:rspec:
  stage: test
  needs: ["mac:build"]

mac:rubocop:
  stage: test
  needs: ["mac:build"]

production:
  stage: deploy

Since the lint job doesn't need anything, it runs instantly, as does linux:build and mac:build. However, if linux:build finishes before mac:build then both linux:rspec and linux:rubocop can start, even before mac:build and the build stage complete.

As usual, without the needs keyword, the production job requires all previous jobs to complete before it starts.

When using needs in your pipelines, you can also view a Directed Acyclic Graph of your jobs in the pipeline view. More on that can be found here: https://docs.gitlab.com/ee/ci/directed_acyclic_graph/index.html

Upvotes: 2

Related Questions