CODEWITHSUNDEEP
gitlabyaml
Joseph Gagnon
Joseph Gagnon

Reputation: 2135

Need to run a GitLab pipeline job only if a prior job succeeds

I have a project in GitLab that has pipeline jobs to build the image, run unit tests and push the built image to a repository.

What needs to happen:

  1. Application is built.
    • If build fails, fail the pipeline. Otherwise, build artifacts are produced.
  2. Unit tests are run.
    • If unit tests fail, fail the pipeline.
    • Make test results (pass/fail) available as artifacts in pipeline.
  3. Built application is pushed to an image repository.
    • Needs build job artifacts.
    • Does not need unit test artifacts.

Pipeline:

stages:
  - build
  - test
  - release

...

build-image:
  stage: build
  extends:
    - .kaniko-build
  variables:
    DOCKERFILE: Dockerfile
    KUBERNETES_MEMORY_REQUEST: 2Gi
    KUBERNETES_MEMORY_LIMIT: 2Gi

run-unit-tests:
  stage: test
  extends:
    - .kaniko-build
  variables:
    DOCKERFILE: Unit-Tests.Dockerfile
    KUBERNETES_MEMORY_REQUEST: 1Gi
    KUBERNETES_MEMORY_LIMIT: 1Gi
  after_script:
    - mkdir -p test/surefire-reports
    - cp -r /app-build/target/surefire-reports ./test
  artifacts:
    when: always
    paths:
      - test/surefire-reports
    reports:
      junit:
        - test/surefire-reports/*.xml
          
push-to-harbor:
  stage: release
  extends:  
    - .release-local-image
  variables:
    SOURCE_IMAGE_NAME: $IMAGE_NAME
    SOURCE_IMAGE_TAG: $CI_PIPELINE_ID
    TARGET_PROJECT: phactory-images
    TARGET_IMAGE_NAME: $IMAGE_NAME
    TARGET_IMAGE_TAG: $CI_COMMIT_REF_SLUG
  needs:
    - build-image

...

What happens:

  1. The build runs successfully.
  2. The unit test and push to harbor jobs execute in parallel. The needs clause of the push to harbor job allows it to run in parallel with the unit test job even though it's in a later stage.
  3. The built application image is pushed to the harbor repository.
  4. Unit tests pass. This job finishes last since it takes longer to run than the push job.

The problem with this is that if the unit test job fails due to a test failure, it does not prevent the built image from being pushed to the repository. This is not what I need.

A suggestion (see prior answer) was made to remove the needs clause from the push to harbor job. This forces the push job to run after the unit test job. If the unit test job fails, the push job does not run. According to the documentation, GitLab by default, flows artifacts from prior jobs to following jobs.

Problem solved, right? Unfortunately, no. If the unit tests pass, the push job executes, but the image that's pushed is what results from running the unit test job Dockerfile. The build job artifacts are nowhere to be found.

I spent some time looking at other attributes of GitLab pipeline jobs and found the dependencies clause (see link below). I've tried this in various ways, but it doesn't seem to do as advertised. It's still pushing the unit test Dockerfile image to the repository and completely ignoring the artifacts of the build job I specified.

push-to-harbor:
  stage: release
  extends:  
    - .release-local-image
  variables:
    SOURCE_IMAGE_NAME: $IMAGE_NAME
    SOURCE_IMAGE_TAG: $CI_PIPELINE_ID
    TARGET_PROJECT: phactory-images
    TARGET_IMAGE_NAME: $IMAGE_NAME
    TARGET_IMAGE_TAG: $CI_COMMIT_REF_SLUG
  dependencies:
    - build-image

Does the dependencies clause not work? Am I using it the wrong way?

GitLab documentation:

https://docs.gitlab.com/ee/ci/yaml/#needs

Use needs to execute jobs out-of-order. Relationships between jobs that use needs can be visualized as a directed acyclic graph.

You can ignore stage ordering and run some jobs without waiting for others to complete. Jobs in multiple stages can run concurrently.

https://docs.gitlab.com/ee/ci/yaml/#dependencies

Use the dependencies keyword to define a list of jobs to fetch artifacts from. You can also set a job to download no artifacts at all.

If you do not use dependencies, all artifacts from previous stages are passed to each job.

Upvotes: 0

Views: 1006

Answers (1)

Egor
Egor

Reputation: 1090

To archive your goal just remove needs from push-to-harbor stage. In this case each job will be dependent on previous jobs and will run only in case if all previous jobs finished successfully. This is default behavior. Also by default all artifacts from previous stages are passed to each job.

Upvotes: 1

Related Questions