CODEWITHSUNDEEP
buildcontinuous-integrationgithub-actionspython-poetrypyproject.toml
Vincent Doba
Vincent Doba

Reputation: 5068

Use precompiled numpy package instead of building it when performing poetry install during build with github actions

I'm using poetry 1.1.12 with python 3.10. My project depends on numpy 1.21.1 that takes 5 minutes to install each time I run my continuous integration pipeline.

Is there a way to make poetry use some kind of compiled numpy package instead of rebuilding it each build ?

I've already mitigated this issue by caching my virtual environment repository following steps described in this answer, but I want a solution that works even if I change my poetry.lock file or if my cache is expired.

I can only use ubuntu-latest image in github actions due to corporate policy rules

My pyproject.toml

[tool.poetry]
name = "test-poetry"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.10"
numpy = "^1.21.1"

[tool.poetry.dev-dependencies]
pytest = "^6.2.5"

[build-system]
requires = ["poetry-core>=1.1.12"]
build-backend = "poetry.core.masonry.api"

My github actions workflow:

name: Continuous Integration

on: push

jobs:
  test-build:
    runs-on: ubuntu-latest

    steps: 
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Install Python 3.10
        uses: actions/setup-python@v2
        with:
          python-version: '3.10'

      - name: Install Poetry packaging manager
        run: curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -

      - name: Configure Poetry
        run: |
          poetry config virtualenvs.create true
          poetry config virtualenvs.in-project true

      - name: Load cached venv
        id: cached-poetry-dependencies
        uses: actions/cache@v2
        with:
          path: .venv
          key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}

      - name: Install project dependencies
        run: poetry install
        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'

      - name: Run test
        run: poetry run pytest

Upvotes: 3

Views: 1587

Answers (1)

Vincent Doba
Vincent Doba

Reputation: 5068

Poetry uses precompiled packages per default if exists. Setting an upper limit on python version made my build use more recent version of numpy that is already precompiled for my version of python

Before numpy 1.21.2, only minimum version of python was set. Numpy 1.21.1 requires python version greater than 3.7

But since numpy 1.21.2, there is also a maximum version of python. Numpy 1.21.2 (to 1.21.5 as time of writing this answer) requires python version greater than 3.7 but strictly lower than 3.11. Here is a summary of numpy/python compatibility:

numpy version python version
1.21.0 Python >=3.7
1.21.1 Python >=3.7
1.21.2 Python >=3.7, <3.11
... ...
1.21.5 Python >=3.7, <3.11

In my pyproject.toml, I've set the python version as follows:

python = "^3.10"

Which conflict with numpy 1.21.2 and greater python version requirements. Thus, poetry decided to install the latest version that is compatible with my python version, which is 1.21.1. But numpy version 1.21.1 is not precompiled for python 3.10, the first numpy version that is precompiled for python 3.10 is numpy 1.21.2.

So every time I installed my poetry project, I would have to rebuild numpy from sources.

To correct this, I changed my pyproject.toml dependencies' section as follows:

[tool.poetry.dependencies]
python = ">=3.10,<3.11"
numpy = "^1.21.5"

And now the poetry install part of my build retrieves the precompiled for python 3.10 numpy version 1.21.5 instead of compiling numpy version 1.21.1, as explained in this answer

Now my poetry install step in my build takes less than 25 seconds instead of 5 minutes.

Upvotes: 2

Related Questions