How can I run a job that doesn't block subsequent stages in Gitlab?

Question

In a project I'm running two stages with these jobs:

• build
  • compile & test
  • generate sonar report
• deploy
  • deploy to staging environment [manual]
  • deploy to production [manual]

The jobs in the deploy stage depend on the outputs of the compile & test job. However the generate sonar report job is not required to finish before I can start any job in the deploy stage. Nevertheless, GitLab insists that all jobs in the build phase have finished before I can launch any job in the deploy phase.

Is there a way I can tell GitLab that the generate sonar report job should not block subsequent pipeline stages? I already tried allow_failure: true on this job but this does not seem to have the desired effect. This job takes a long time to finish and I really don't want to wait for it all the time before being able to deploy.

Answer 1

From my point of view, it depends on your stage semantics. You should try to decide what is mostly important in your pipeline: clarity on stages or get the job done.

GitLab has many handy features like needs keyword you can use it to specify direct edges on the dependency graph.

stages:
    - build
    - deploy

build:package:
    stage: build
    script: 
        - echo "compile and test"
        - mkdir -p target && echo "hello" > target/file.txt
    artifacts:
        paths: 
            - ./**/target

build:report:
    stage: build
    script:
        - echo "consume the target artifacts"
        - echo "waiting for 120 seconds to continue"
        - sleep 120
        - mkdir -p target/reports && echo "reporting" > target/reports/report.txt
    artifacts:
        paths: 
            - ./**/target/reports

deploy:
    stage: deploy
    needs: ["build:package"]
    script:
        - echo "deploy your package on remote site"
        - cat target/file.txt

Answer 2

We have similar situation, and while we do use allow_failure: true, this does not help when the Sonar job simply takes a long time to run, whether it fails or succeeds.

Since you are not wanting your deploy stage to actually be gated by the outcome of the generate sonar report job, then I suggest moving the generate sonar report job to the deploy stage, so your pipeline would become:

• build
  • compile & test
• deploy
  • deploy to staging environment [manual]
  • deploy to production [manual]
  • generate sonar report [allow_failure: true]

This way the generate sonar report job does not delay your deploy stage jobs

The other benefit of running generate sonar report after build & test is that you can save coverage reports from the build & test job as Gitlab job artifacts, and then have generate sonar report job consume them as dependencies, so Sonar can monitor your coverage, too

Finally, we find it useful to separate build & test into build, then test, so we can separate build failures from test failures - and we can then also run multiple test jobs in parallel, all in the same test stage, etc. Note you will need to convey the artifacts from the build job to the test job(s) via Gitlab job artifacts & dependencies if you choose to do this

Answer 3

Unless I'm mistaken, this is currently not possible, and there is an open feature proposal, and another one similar to add what you are suggesting.