CODEWITHSUNDEEP
pythongithubpipgithub-actions
rcb
rcb

Reputation: 73

Issue caching python dependencies in GitHub Actions

I have the following steps in a github action:

steps:
      - name: Check out repository code
        uses: actions/checkout@v2

      - name: Cache dependencies
        id: pip-cache
        uses: actions/cache@v2
        with:
          path: ~.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-

      - name: Install dependencies
        if: steps.pip-cache.outputs.cache-hit != 'true'
        run: pip install -r requirements.txt
     
      - name: run mypy
        run: mypy .

The caching works fine, but when a cache hit occurs, and I try to run mypy, it fails with:

Run mypy .
/home/runner/work/_temp/9887df5b-d5cc-46d7-90e1-b884d8c49272.sh: line 1: mypy: command not found
Error: Process completed with exit code 127.

The whole point of caching dependencies is so I don't have to install them every time I run the workflow. How do I use the cached dependencies?

Upvotes: 5

Views: 2968

Answers (1)

Cecil Curry
Cecil Curry

Reputation: 10226

You're only caching source tarballs and binary wheels downloaded by pip. You're not caching:

  • Installed Python packages (i.e., the site-packages/ subdirectory of the active Python interpreter).
  • Installed entry points (i.e., executable commands residing in the current ${PATH}).

That isn't necessarily a bad thing. Merely downloading assets tends to consume a disproportionate share of scarce GitHub Actions (GA) minutes; caching assets trivially alleviates that issue.

In other words, remove the if: steps.pip-cache.outputs.cache-hit != 'true' line to restore your GitHub Actions (GA) workflow to sanity.

But... I Want to Cache Installed Packages!

Challenge accepted. This is feasible – albeit more fragile. I'd advise just caching pip downloads unless you've profiled the pip install command to be a significant installation bottleneck.

Let's assume that you still want to do this. In this case, something resembling the following snippet should get you where you want to go:

  - uses: 'actions/setup-python@v2'
    with:
      # CAUTION: Replace this hardcoded "3.7" string with the
      # major and minor version of your desired Python interpreter.
      python-version: "3.7"
  - uses: 'actions/cache@v2'
    id: cache
    with:
      # CAUTION: Replace this hardcoded "python3.7" dirname with
      # the dirname providing your desired Python interpreter.
      path: ${{ env.pythonLocation }}/lib/python3.7/site-packages/*
      key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
      restore-keys: |
        ${{ runner.os }}-pip-
        ${{ runner.os }}-

As noted, you'll want to manually replace the hardcoded 3.7 and python3.7 substrings above with something specific to your use case. That's one of several reasons why this is dangerously fragile. Another is that the ${{ env.pythonLocation }} environment variable set by the setup-python GitHub Action has been infamously undocumented for several years. </gulp> (env.pythonLocation documentation was added from v4.0.0.)

In theory, adding the above directly under your existing uses: actions/cache@v2 list item should suffice. Good luck and Godspeed as you travel into the terrifying unknown.

Upvotes: 8

Related Questions