CODEWITHSUNDEEP
gitlabgitlab-ci
Udo G
Udo G

Reputation: 13111

How to access multiple repositories in CI build?

We have a project that is composed of multiple (non-public) repositories.

To build the whole project, the build system needs to have the files of all repositories (master branches).

Is there a way I can configure GitLab CI to provide the repositories I need?

I guess I could do a git fetch or similar during the CI build, but how to deal with authentication then?

Upvotes: 121

Views: 160835

Answers (6)

MGamarra
MGamarra

Reputation: 1

It is also possible to add the submodule directly in the job code, as shown in the example below.

build:
  stage: build
  before_script:
    - git config --global http.sslVerify "false"
    - git submodule add -b main --force [email protected]/group/project.git     
    - git submodule init
    - git submodule update --recursive --remote
  script:
    - ls -lha

Upvotes: 0

microe14
microe14

Reputation: 2921

If you are running GitLab version 8.12 or later (released September 2016), the permissions model was reworked. Along with this new permission model comes the CI environment variable CI_JOB_TOKEN. The premium version of GitLab uses this environment variable for triggers, but you can use it to clone repos.

dummy_stage:
  script:
    - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.instance/group/project.git

If it is another repository, you might need to grant the CI that will be cloning access.

In Gitlab 16.11:

  1. goto Settings > CI/CD (of the foreign repo)
    eg https://gitlab.instance/group/project/-/settings/ci_cd
  2. goto Token Access
  3. add the repo you want to checkout from

Upvotes: 274

Bomi Chen
Bomi Chen

Reputation: 122

if you're using fastlane to build iOS app, you can succeed by change git_url in Matchfile.

Steps:

  1. add personal access token in gitlab account: User Setting -> Access Tokens
  2. put the access token in fastlane Match file.

ex:

git_url("https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/<namespace>/<project>")

ref: https://docs.gitlab.co.jp/ee/ci/jobs/ci_job_token.html

Upvotes: 1

mangelajo
mangelajo

Reputation: 31

In my use case, I am pulling git sub-modules with LFS storage; in this case, this is what worked best:

sed -i "s%url = [email protected]:%url = https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/%g" .git/config .gitmodules

Upvotes: 2

Ivan
Ivan

Reputation: 9695

A couple of workarounds (I hate this word!) that work-around-ed for me:

  1. Using git submodule, see https://docs.gitlab.com/ce/ci/git_submodules.html

  2. Re-using $CI_REPOSITORY_URL defined by Gitlab and available even inside child Docker containers. This env var already contains username and password, that can be used for another repo on the same server. See snippet from .gitlab-ci.yml:

- BASE_URL=`echo $CI_REPOSITORY_URL | sed "s;\/*$CI_PROJECT_PATH.*;;"`
- REPO_URL="$BASE_URL/thirdparty/gtest.git"
- REPO_DIR=thirdparty/gtest
- rm -fr $REPO_DIR
- git clone $REPO_URL $REPO_DIR
  1. Even storing that URL with username\password in ~/.git-credentials file and configuring git to use it via credential.helper. All further "git clone" commands will use it.
- echo Storing git credentials to be used by "git clone" commands without username and password ...
- GIT_CREDENTIALS_FILE=~/.git-credentials
- BASE_URL=`echo $CI_REPOSITORY_URL | sed "s;\/*$CI_PROJECT_PATH.*;;"`
- echo $BASE_URL > $GIT_CREDENTIALS_FILE
- git config --global credential.helper store --file=$GIT_CREDENTIALS_FILE

HOWEVER !

Having spent quite a few years in CI \ CD field, I don't think it's a good design that requires linking repositories as sources.

Yes, in classic CI tools like Jenkins or TeamCity you can create a job that fetches several Git repos in different subdirectories.

But I like GitLab CI way of Pipeline As Code, where .gitlab-ci.yml controls the build of this very repo and you don't even have to think about that whole pre-build step of getting sources. Then such build would publish binary artifacts and downstream projects\repos can use those instead of sources of dependencies. It's also faster.

Separation of concerns.

I don't there is an official way in my .gitlab-ci.yml to use artifacts of another project. But there are other ways like hooks, Gitlab API, though such bespoke solutions require maintenance.

There's better way - to publish\fetch artifacts to\from external widely-adopted Package Manager. Depending on your language it could be Maven, NuGet, npm, jFrog Artifactory, Nexus, etc. Another advantage of this method is that developers can follow the same process in their local builds, which is not easily done if dependencies are defined in .gitlab-ci.yml

It's a bigger problem for native code (Cxx) mainly due to Binary Interface compatibility, but things like Conan.io etc are slowly catching up.

Upvotes: 34

Drew Blessing
Drew Blessing

Reputation: 2705

You can add a deploy key to all projects. Then configure the deploy key's private key on the runner(s). Use normal git commands in your build process to clone the repositories on the runner. This may require a bit of configuration on your runners, but it should work.

You can either generate one SSH key pair and use that on all runners or generate one key pair per runner. To generate an SSH key pair follow SSH key documentation. The private key should be placed in 'gitlab-runner' user's .ssh directory so the git command can present it at clone time. The public key should be added to the GitLab project as a deploy key. Add a deploy key in the project settings -> 'Deploy Keys'.

Upvotes: 8

Related Questions