CODEWITHSUNDEEP
pythonpipgitlabcontinuous-integrationgitlab-ci
kevinlinxc
kevinlinxc

Reputation: 604

Caching two different versioned Python venvs in GitLab CI

The first part of my GitLab CI has to run with Python 3.7, with certain packages, and the second part with Python 3.6, with different packages.

I'm wondering if someone could provide an example of how to use the cache feature so that the installation of packages during both stages uses the cache.

In other words, I want it so that cache1 is created on the first run of the Python 3.7 stage, and cache2 is created on the first run of the Python 3.6 stage, and then each stage uses its respective cache in future runs.

Upvotes: 1

Views: 870

Answers (2)

Lei Yang
Lei Yang

Reputation: 4335

Your question is not very clear, I assume by cache you mean the installed packages by pip install.
I don't think there's a convenient way to share libraries among different python versions.
Paticularly in gitlab ci, if you have same python version in multiple jobs, I think reuse cache is possible, however, I prefer using prebuilt docker images with packages installed, because it build once and run everywhere. pip install in pipeline will waste cpu/network resource in every run.

your desired way

image: python3.6

job1:
  script:
    - pip install a
    - assert import a success
  cache:
    key: packages
    paths:
      - /pythoninstallpath/lib/**/*

job2:
  script:
    - pip install b
    - assert import a,b success
  cache:
    key: packages
    paths:
      - /pythoninstallpath/lib/**/*

my suggest way



job1:
  image: python3.6 with a installed
  script:
    - assert import a success

job2:
  image: python3.6 with a,b installed
  script:
    - assert import a,b success

Upvotes: 0

sytech
sytech

Reputation: 40921

As long as the two jobs use different values for cache:key:, the caches will be separate for their respective jobs.

One easy way to do this is by using $CI_JOB_NAME in the key. It's often common to also keep caches per-branch, using $CI_COMMIT_REF_SLUG.

.cache_python:
  variables:
    FF_USE_FASTZIP: 1  # enable faster caching/artifacting
    PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip_cache"
  cache:
    key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
    paths:  # cache the venv and pip cache (you may opt to use just 1 of these)
      - venv
      - $PIP_CACHE_DIR

build/a:
  image: python:3.6-slim
  extends: ".cache_python"
  script:
    - python -m venv venv
    - source venv/bin/activate
    - pip install ...
    - make build

build/b:
  image: python:3.7-slim
  extends: ".cache_python"
  script:
    - python -m venv venv
    - source venv/bin/activate
    - pip install ...
    - make build

Both build/a and build/b will have independent caches that will be reused across pipelines on the same branch.

See more in common use cases for caches including how to share caches in different scenarios.

Upvotes: 1

Related Questions