CODEWITHSUNDEEP
gitlab-ci
ZioByte
ZioByte

Reputation: 2994

How to define a GitLab CI job to depend on either one or one another previous job?

I want to define a pipeline to compile, deploy to target and test my project.

This should happen in two distinct ways: an incremental (hopefully fast) build at each commit and a full build scheduled at night.

The following .gitlab-ci.yml has all jobs marked "manual" for testing purposes.

stages:
    - build
    - deploy
    - test

variables:
    BUILD_ARTIFACTS_DIR: "artifacts"

build-incremental:
    timeout: 5h 
    stage: build
    script:
        - echo "Building"
        - ./ci/do-prep
        - echo "done."
    artifacts:
        paths:
            - $BUILD_ARTIFACTS_DIR/
    variables:
        BUILD_TOP_DIR: "/workspace/builds"
    tags:
        - yocto
    when: manual

build-nightly:
    timeout: 5h
    stage: build
    script:
        - echo "Building"
        - ./ci/do-prep
        - echo "done."
    artifacts:
        paths:
            - $BUILD_ARTIFACTS_DIR/
    tags:
        - yocto
    when: manual

deploy:
    stage: deploy
    script:
        - echo "Deploying..."
        - ./ci/do-deploy
        - echo "done."
    tags:
        - yocto
    dependencies:
        - build
    when: manual

test:
    stage: test
    script:
        - echo "Testing..."
        - ./ci/do-test
        - echo "done."
    tags:
        - yocto
    dependencies:
        - deploy
    when: manual

This fails with message: deploy job: undefined dependency: build.

Cow do I explain to GitLab deploy stage needs either build-incremental or build-nightly artifacts?

Later I will have to understand how to trigger build-incremental at commit and build-nightly using a schedule, but that seems to be a different problem.

Upvotes: 25

Views: 73614

Answers (3)

zulucoda
zulucoda

Reputation: 858

The best way to accomplish this is to use needs more info on needs

Example below:

stages:
  - 📼 start-main-job
  - 📎 start-main-job-2


DEV before-main-job:
  stage: 📼 start-main-job
  needs: []
  when: manual

DEV start-main-job:
  stage: 📼 start-main-job
  needs: ["DEV before-main-job"]
  when: on_success

DEV before-main-job-2:
  stage: 📎 start-main-job-2
  needs: []
  when: manual

DEV start-main-job-2:
  stage: 📎 start-main-job-2
  needs: ["DEV before-main-job-2"]
  when: on_success

Example pipeline

enter image description here

Upvotes: 0

Adam Marshall
Adam Marshall

Reputation: 7649

For your scenario, there are two separate paths through your pipeline depending on the "source": a 'push' or a 'schedule'. You can get the source of the pipeline with the CI_PIPELINE_SOURCE variable. I build these paths separately at first, then combine them:

# First source: push events
stages:
  build
  deploy
  test

variables:
    BUILD_ARTIFACTS_DIR: "artifacts"

build-incremental:
    timeout: 5h 
    stage: build
    script:
        - echo "Building"
        - ./ci/do-prep
        - echo "done."
    artifacts:
        paths:
            - $BUILD_ARTIFACTS_DIR/
    variables:
        BUILD_TOP_DIR: "/workspace/builds"
    tags:
        - yocto
    rules:
      - if: $CI_PIPELINE_SOURCE == 'push'
        when: manual
      - when: never

deploy-incremental:
    stage: deploy
    script:
        - echo "Deploying..."
        - ./ci/do-deploy
        - echo "done."
    tags:
        - yocto
    needs: ['build-incremental']
    rules:
      - if $CI_PIPELINE_SOURCE == 'push'
        when: always
      - when: never

test-incremental:
    stage: test
    script:
        - echo "Testing..."
        - ./ci/do-test
        - echo "done."
    tags:
        - yocto
    needs: ['deploy-incremental']
    rules:
      - if: $CI_PIPELINE_SOURCE == 'push'
        when: always
      - when: never

In this path, if the source is a push, the build step will run upon manual input, otherwise it will never run. Then, the deploy-incremental step will run automatically (without waiting for other jobs or stages) as long as the source is a push, otherwise it will never run. Finally the test-incremental job will run automatically without waiting for other jobs or stages if it's a push like above.

Now we can build the schedule path:

# Scheduled path:

stages:
  build
  deploy
  test

variables:
    BUILD_ARTIFACTS_DIR: "artifacts"

build-schedule:
    timeout: 5h 
    stage: build
    script:
        - echo "Building"
        - ./ci/do-prep
        - echo "done."
    artifacts:
        paths:
            - $BUILD_ARTIFACTS_DIR/
    variables:
        BUILD_TOP_DIR: "/workspace/builds"
    tags:
        - yocto
    rules:
      - if: $CI_PIPELINE_SOURCE === 'schedule'
        when: manual
      - when: never

deploy-schedule:
    stage: deploy
    script:
        - echo "Deploying..."
        - ./ci/do-deploy
        - echo "done."
    tags:
        - yocto
    needs: ['build-schedule']
    rules:
      - if $CI_PIPELINE_SOURCE == 'schedule'
        when: always
      - when: never

test-schedule:
    stage: test
    script:
        - echo "Testing..."
        - ./ci/do-test
        - echo "done."
    tags:
        - yocto
    needs: ['deploy-schedule']
    rules:
      - if: $CI_PIPELINE_SOURCE == 'schedule'
        when: always
      - when: never

This works the same way as the push path, but we check to see if the source is schedule.

Now we can combine the two paths:

Combined result:
stages:
  build
  deploy
  test

variables:
    BUILD_ARTIFACTS_DIR: "artifacts"

build-incremental:
    timeout: 5h 
    stage: build
    script:
        - echo "Building"
        - ./ci/do-prep
        - echo "done."
    artifacts:
        paths:
            - $BUILD_ARTIFACTS_DIR/
    variables:
        BUILD_TOP_DIR: "/workspace/builds"
    tags:
        - yocto
    rules:
      - if: $CI_PIPELINE_SOURCE == 'push'
        when: manual
      - when: never

build-schedule:
    timeout: 5h 
    stage: build
    script:
        - echo "Building"
        - ./ci/do-prep
        - echo "done."
    artifacts:
        paths:
            - $BUILD_ARTIFACTS_DIR/
    variables:
        BUILD_TOP_DIR: "/workspace/builds"
    tags:
        - yocto
    rules:
      - if: $CI_PIPELINE_SOURCE == 'schedule'
        when: manual
      - when: never

deploy-incremental:
    stage: deploy
    script:
        - echo "Deploying..."
        - ./ci/do-deploy
        - echo "done."
    tags:
        - yocto
    needs: ['build-incremental']
    rules:
      - if $CI_PIPELINE_SOURCE == 'push'
        when: always
      - when: never

deploy-schedule:
    stage: deploy
    script:
        - echo "Deploying..."
        - ./ci/do-deploy
        - echo "done."
    tags:
        - yocto
    needs: ['build-schedule']
    rules:
      - if $CI_PIPELINE_SOURCE == 'schedule'
        when: always
      - when: never

test-incremental:
    stage: test
    script:
        - echo "Testing..."
        - ./ci/do-test
        - echo "done."
    tags:
        - yocto
    needs: ['deploy-incremental']
    rules:
      - if: $CI_PIPELINE_SOURCE == 'push'
        when: always
      - when: never

test-schedule:
    stage: test
    script:
        - echo "Testing..."
        - ./ci/do-test
        - echo "done."
    tags:
        - yocto
    needs: ['deploy-schedule']
    rules:
      - if: $CI_PIPELINE_SOURCE == 'schedule'
        when: always
      - when: never

A pipeline like this is tedious and takes a bit to build, but works great when you have multiple paths/ways to build the project.

Upvotes: 7

William Arias
William Arias

Reputation: 199

To tell Gitlab that your deploy stage needs certain artifacts from a specific job: Try naming dependencies by job name. In deploy you are defining a dependency with build which is a stage name not the one of the job you want to pick the artifact. Example:

deploy:
stage: deploy
script:
    - echo "Deploying..."
    - ./ci/do-deploy
    - echo "done."
tags:
    - yocto
dependencies:
    - build-incremental
when: manual

more info and examples here dependencies

Upvotes: 6

Related Questions