CODEWITHSUNDEEP
pythongitlabcontinuous-integrationpython-poetry
Angel
Angel

Reputation: 2865

How to cache "poetry install" for Gitlab CI?

Is there a way to cache poetry install command in Gitlab CI (.gitlab-ci.yml)?

For example, in node yarn there is a way to cache yarn install (https://classic.yarnpkg.com/lang/en/docs/install-ci/ Gitlab section) this makes stages a lot faster.

Upvotes: 15

Views: 11552

Answers (3)

Richard Scholtens
Richard Scholtens

Reputation: 1023

Combining the answers of @Chris and @Sean Connolly helped me to transfer the dependencies of my Python virtual environment and also the poetry binary folder.

I used the POETRY_VIRTUALENVS_PATH to specify where the virtual environment will be instantiated. The poetry documentation states in step 2 Install Poetry (advanced) from the With the official installer that it is possible to define a custom install folder. Then I install poetry and specify that it should be installed in %CI_PROJECT_DIR/poetry. Now I know where poetry and the venv folders are installed thus I can passed to the next stage using artifacts. In this stage I export the poetry binary so it can be used without referencing the folder directly.

The yaml file below should work:

---
stages:
  - lint
  - test

pre-commit:
  stage: lint
  image: python:3.11-alpine
  variables:
    POETRY_VIRTUALENVS_PATH: $CI_PROJECT_DIR/.venv
  before_script:
    - apk add curl
    - apk add git
    - curl -sSL https://install.python-poetry.org | POETRY_HOME=$CI_PROJECT_DIR/poetry python3 -
    - export PATH="/builds/GROUP_NAME/REPOSITORY_NAME/poetry/bin:$PATH"
    - poetry config virtualenvs.in-project true
    - poetry install
  script:
    - poetry run pre-commit run --all-files
  artifacts:
    paths:
      - $CI_PROJECT_DIR/.venv
      - $CI_PROJECT_DIR/poetry


quality-checks:
  stage: test
  image: python:3.11-alpine
  dependencies:
    - pre-commit
  before_script:
    - apk add curl
    - apk add bash
    - apk add git
  script:
    - export PATH="/builds/GROUP_NAME/REPOSITORY_NAME/bin:$PATH"
    - source $CI_PROJECT_DIR/.venv/bin/activate
    - poetry run pytest

Upvotes: 1

Sean Connolly
Sean Connolly

Reputation: 5803

You can set the Poetry virtualenv cache directory to something within the GitLab project space, e.g.:

variables:
    POETRY_VIRTUALENVS_PATH: $CI_PROJECT_DIR/venv

and then cache it like normal:

cache:
    paths:
        - venv

Upvotes: 2

Chris
Chris

Reputation: 136958

GitLab can only cache things in the working directory and Poetry stores packages elsewhere by default:

Directory where virtual environments will be created. Defaults to {cache-dir}/virtualenvs ({cache-dir}\virtualenvs on Windows).

On my machine, cache-dir is /home/chris/.cache/pypoetry.

You can use the virtualenvs.in-project option to change this behaviour:

If set to true, the virtualenv wil be created and expected in a folder named .venv within the root directory of the project.

So, something like this should work in your gitlab-ci.yml:

before_script:
  - poetry config virtualenvs.in-project true

cache:
  paths:
    - .venv

Upvotes: 21

Related Questions