CODEWITHSUNDEEP
pipdockerfilegithub-actions
tgk
tgk

Reputation: 4096

Caching pip dependencies in docker builder with github actions

I'm using github actions to build a docker image. I cache the docker layers (which works well) but sometimes layers aren't reused (not sure why! i'm following best practices but ignore that aside) and I'm using a fallback pip cache to speed up the build.

Dockerfile is pretty simple:

# syntax=docker/dockerfile:1.2
FROM python:3.10-slim
RUN python -m pip install --upgrade pip &&  pip cache dir
COPY requirements.txt ./
RUN --mount=type=cache,target=/root/.cache/pip/ pip install -r requirements.txt

I'm using the --mount flag from syntax 1.2 which "allows the build container to cache directories for compilers and package managers"

Github actions cache (which i'd like to mount on the build container):

- name: Cache pip
  uses: actions/cache@v3
  with:
    path: /root/.cache/pip
    key: pip-{{ hashFiles('requirements*.txt') }}

and docker build

- name: Build and push
  uses: docker/build-push-action@v2
  with:
    cache-from: type=gha
    cache-to: type=gha,mode=max

How do I mount the pip cache into the docker builder container?

Right now the cache is empty using this command in the dockerfile:

RUN --mount=type=cache,target=/root/.cache/pip/ pip cache info && ls /root/.cache/pip

outputs:

19 RUN --mount=type=cache,target=/root/.cache/pip/ pip cache info && ls /root/.cache/pip
#19 0.387 Package index page cache location: /root/.cache/pip/http
#19 0.387 Package index page cache size: 0 bytes
#19 0.387 Number of HTTP files: 0
#19 0.387 Wheels location: /root/.cache/pip/wheels
#19 0.387 Wheels size: 0 bytes
#19 0.387 Number of wheels: 0
#19 DONE 0.4s

Upvotes: 5

Views: 3369

Answers (1)

tgk
tgk

Reputation: 4096

So this works actually and dropped my install time from 180s to 30-40s it's a little funky but I got it working for both pip and yarn (yarn doesn't save anytime tho since I'm using yarn 1 and most of the time is spent on I/O reading writing)

You'll need buildx v0.8.0+ to support --build-context and you'll need docker frontend syntax v1.4+ to support RUN --mount I also use github actions cache to handle installing and caching pip dependencies. It looks like this

  1. github cache action, install pip requirements, also get pip cache dir: ::set-output name=pip-dir::$(pip cache dir) to use in next step
  2. I'm using docker build and push action, which supports --build-context:
    build-contexts: pip_cache=${{steps.vars.outputs.pip-dir}}
  3. Update the docker file to use the pip_cache build context RUN --mount=type=cache,target=/root/.cache/pip,from=pip_cache pip install -r requirements.txt

On build pip install using cached packages:

#22 RUN --mount=type=cache,target=/root/.cache/pip,from=pip_cache pip install -r requirements.txt
#22 0.587 Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (0.34.2)
#22 0.592 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#22 1.440 Collecting aenum==2.2.3
#22 1.442   Using cached aenum-2.2.3-py3-none-any.whl (40 kB)
#22 1.495 Collecting alembic==1.7.3
#22 1.498   Using cached alembic-1.7.3-py3-none-any.whl (208 kB)

It's mostly worth it for the time saved building wheels, but downloads are pretty snappy regardless.

Upvotes: 4

Related Questions