CODEWITHSUNDEEP
gitpipgitlab-ci
Jean-Baptiste
Jean-Baptiste

Reputation: 967

Pip install a private repo from Gitlab with Personal Access Token on Gitlab-CI

I'm using Gitlab to hosts two private Python packages. One, say package B, depends on the other, say A.

I would like to setup correctly package B, so I have a setup.py with

install_requires=[
    'packagea @ git+https://gitlab.com/me/packagea.git',
    ...
]

as well as a requirement.txt containing

git+https://gitlab.com/me/[email protected]

And that works well when a user installs package B with pip install git+https://gitlab.com/me/packageb.git, or when a user clones package B and then run pip install -r requirements.txt.

However, I would like to setup continuous integration on package B. My gitlab-ci.yml looks like

image: python:3.7
before_script:
  - pip install -r requirements.txt

pylint:
  script:
    - pylint **/*.py

This fails because the Gitlab-CI runner does not have any username/password to authenticate and pull package A repo. I've read that I could generate a Personal Access Token for the CI runner to authenticate, but this would mean that setup.py and requirements.txt contain the token. This seems ugly to me (I'd like the other users to still use their own username/password).

How can I achieve this?

Upvotes: 8

Views: 15510

Answers (4)

Phrogz
Phrogz

Reputation: 303253

With command-line access, e.g.

pip3 install -U git+https://gitlab.mycompany.com/foo/bar.git

using a personal access token, when you are prompted for a username, you enter any (non-empty) string. Then, when you are prompted for a password for that username, enter your personal access token.

Upvotes: 0

Matheus Gagno Brunetti
Matheus Gagno Brunetti

Reputation: 792

In place of personal access token, I figured it out with the generated token when a CI Pipeline is triggered.

In this way, I solved this problem by using the user from pipeline (gitlab-ci-token) and also the token from pipeline (CI_JOB_TOKEN).

Then, inside the Dockerfile, I added the following commands before the pip install -r requirements.

... 

ARG CI_JOB_TOKEN 
RUN sed -i 's/<GITLAB_URL>/gitlab-ci-token:${CI_JOB_TOKEN}@<GITLAB_URL>/g' requirements.txt

...

After that I build the container in the .gitlab-ci.yml with the following command.

docker build . --build-arg CI_JOB_TOKEN=${CI_JOB_TOKEN} -t <REGISTRY_URL>:<TAG>

I hope that can help someone!

Upvotes: 4

larsyunker
larsyunker

Reputation: 131

In case anyone is like me and gets lost trying to address this pipeline issue, the solution that helped me was this Gitlab issue and the discussion below.

To summarize:

In .gitlab-ci.yml add these lines to your before_script section:

before_script
  - ...
  - echo "machine gitlab.com" > ~/.netrc
  - echo "login gitlab-ci-token" >> ~/.netrc
  - echo "password ${CI_JOB_TOKEN}" >> ~/.netrc
  - if [ -f "requirements.txt" ]; then pip install -r requirements.txt ; fi

In your requirements.txt file, specify the private dependency with the name, url, and egg (the egg must be the package name). e.g.

thispackageisonpip>=1.2.3
soisthisone==1.0.0
...
privatepackage @ git+https://gitlab.com/user/repository.git@tag#egg=privatepackage

where tag is e.g. 'v1.3.3' (note that there must be a tagged version to check out).

The private package should now be checked out successfully by the pipeline. I haven't been able to get this to work where requirements are specified in install_requires of setup.py, but at least this is workable.

Upvotes: 13

Lisa
Lisa

Reputation: 204

Since package A is a private repo, you will need permissions to access it in any scenario.

You might consider using GitLab Deploy Keys. That way, any user (including the ci) with the deploy key can read or write to the repo (depending on the access you give the key). This ties the key to the repo, rather than to your user account.

It is important to note that this deploy key gives access to the private repo, so it is a secret and should be treated as such.

Secrets management is a complex, subjective topic with solutions varying significantly based on many factors, including how secure you need your project to be, ease of use, and team size. A common way to avoid storing passwords in plaintext is to use custom environment variables. If you are looking for recommendations on how to set up a project this way, this post might be a good place to start.

Upvotes: 2

Related Questions