CODEWITHSUNDEEP
gitlabgitlab-ci
Anton Menshov
Anton Menshov

Reputation: 2326

Moving scripts into a separate file in gitlab-ci.yml to avoid code duplication and including it from several files

I am trying to set up a CI with minimal code duplication using .gitlab-ci.yml.

With that, I am separating the configuration in separate files and reusing parts of it that are common.

I have a separate repository with Gitlab CI settings: gitlab-ci and several projects that use it to form their own CI pipelines.

Contents of gitlab-ci repository

template_jobs.yml:

.sample:
  rules:
    -  if: '$CI_PIPELINE_SOURCE == "push"'
       when: on_success
    -  when: never

jobs_architectureA.yml:

include:
  - local: '/template_jobs.yml'

.script_core: &script_core
  -  echo "Running stage"

test_archA:
  extends:
    - .sample
  stage: test
  tags:
    - architectureA
  script:
    - *script_core

jobs_architectureB.yml:

include:
  - local: '/template_jobs.yml'

.script_core: &script_core
  -  echo "Running stage"

test_archB:
  extends:
    - .sample
  stage: test
  tags:
    - architectureB
  script:
    - *script_core

Project with code contents:

In the actual project (separate repositories per project, and I have a lot of them), I have the following:

.gitlab-ci.yml:

stages:
  - test

include:
  - project: 'gitlab-ci'
    file: '/jobs_architectureA.yml'
  - project: 'gitlab-ci'
    file: '/jobs_architectureB.yml'

This configuration works fine and allows to include only some architectures for some modules while sharing rules between the job templates.

However, it's easy to notice one code duplication: both jobs_architectureA.yml and jobs_architectureB.yml contain a common section:

.script_core: &script_core
  -  echo "Running stage"

It would be ideal to move it into a separate file: template_scripts.yml and include from both jobs_architectureA.yml* and jobs_architectureB.yml. However, that results in the invalid YAML (at least from Gitlab's point of view).

With that, I make a conclusion that I can share the rules as the mechanism of their usage is via extends keyword; however, I am not able to do it with the scripts: as it uses &/* anchoring mechanic on the YAML level.

Ideally, I want something along the lines of:

Contents of the ideal (conceptually) gitlab-ci repository

template_jobs.yml:

.sample:
  rules:
    -  if: '$CI_PIPELINE_SOURCE == "push"'
       when: on_success
    -  when: never

template_scripts.yml:

.script_core: &script_core
  -  echo "Running stage"

jobs_architectureA.yml:

include:
  - local: '/template_jobs.yml'
  - local: '/template_scripts.yml'

test_archA:
  extends:
    - .sample
  stage: test
  tags:
    - architectureA
  script:
    - *script_core  # this becomes invalid, as script_core is in the other file, even though it is included at the top

jobs_architectureB.yml:

include:
  - local: '/template_jobs.yml'
  - local: '/template_scripts.yml'

test_archB:
  extends:
    - .sample
  stage: test
  tags:
    - architectureB
  script:
    - *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top

Note, while this might not look like a big deal, in reality, I have many more pieces to the scripts, and the actual script is much larger. Thus, currently, it is duplicated code all over the place which is very prone to mistakes.

Upvotes: 4

Views: 9118

Answers (3)

tosho-ait
tosho-ait

Reputation: 1

The extends: keyword can be used with multiple jobs to extend from (on recent versions of GitLab). You can simply do the following:

include:
  - local: '/template_jobs.yml'

.script_core:
  -  echo "Running stage"
 
test_archA:
  extends:
    - .sample
    - .script_core
  stage: test
  tags:
    - architectureA

And after that move the .script_core to an import.

Upvotes: 0

VonC
VonC

Reputation: 1330092

in reality, I have many more pieces to the scripts, and the actual script is much larger

Adding to Cyril's solution, GitLab 13.12 (May 2021) can help scale those includes:

Support wildcards when including YAML CI/CD configuration files

The includes: keyword for CI/CD pipelines lets you break one long .gitlab-ci.yml file into multiple smaller files to increase readability.
It also makes it easier to reuse configuration in multiple places.

Frequently there are multiple files included into a single pipeline, and they all might be stored in the same place.

In this release, we add support to use the * wildcard with the local includes: keyword. You can now make your includes: sections more dynamic, less verbose, and easier to read, check out how we are dogfooding it in GitLab.

https://about.gitlab.com/images/13_12/wildcard1.png -- Support wildcards when including YAML CI/CD configuration files

See Documentation and Issue.

Upvotes: 2

Cyril Jouve
Cyril Jouve

Reputation: 1050

my solution is to not include template_jobs.yml and template_scripts.yml directly in jobs_architectureA.yml but only in the "final" .gitlab-ci.yml

taking you exemple, /template_jobs.yml//template_scripts.yml do not change.

jobs_architectureA.yml loses the include:

test_archA:
  extends:
    - .sample
  stage: test
  tags:
    - architectureB
  script:
    - *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top

and .gitlab-ci.yml becomes:

stages:
  - test

include:
  - local: '/template_jobs.yml'
  - local: '/template_scripts.yml'
  - project: 'gitlab-ci'
    file: '/jobs_architectureA.yml'
  - project: 'gitlab-ci'
    file: '/jobs_architectureB.yml'

Upvotes: 4

Related Questions