python-packaging: share build-backend between requirements and my package

Question

I have several dependecies for my python pyackage. Two of those requirements are prophet (https://github.com/facebook/prophet) and george (https://github.com/dfm/george). The problem with those packages is, that they require installing numpy, pybind11 and other packages before the installation since they are used during the build process. However, I want to make it as easy as possible for the user without prior installation of several packages.

Packages that are required for building a package can be specified with setup_requires in setup.py or in [build-system] inside pyproject.toml. I thought I would be able to achieve a single command installation with "pip install" by specyfing those build requirements of the external package in my pyproject.toml. However, this doesn´t work. My suggestion is, that for each package for which a wheel has to be build a separate build-backend is used. Therefore the build requirements aren´t located when building the third party package.

I found almost no documentation for build-backend and build requirements. Can you anyone give me a hint?

I have a minimum working example here: https://github.com/FelixKleineBoesing/pypackagingdemo With docker installed you can start the building process with:

docker build .

Without line 10, pip isn´t able to successfully build george since numpy and pybind11 are missing.

Full traceback:

Sending build context to Docker daemon  81.41kB
Step 1/8 : FROM python:3.7
 ---> 34a518642c76
Step 2/8 : RUN mkdir /buildbackend
 ---> Running in 38e398f63eed
Removing intermediate container 38e398f63eed
 ---> f88b7b3f4056
Step 3/8 : COPY ./abctest /buildbackend/abctest
 ---> 5976a8f6d708
Step 4/8 : COPY ./pyproject.toml /buildbackend/pyproject.toml
 ---> 73d4bb506910
Step 5/8 : COPY ./setup.py /buildbackend/setup.py
 ---> 0a377986a29d
Step 6/8 : COPY ./Manifest.in /buildbackend/Manifest.in
 ---> c30e903f8f84
Step 7/8 : RUN cd /buildbackend && pip3 install .
 ---> Running in 7925fc1a9a45
Processing /buildbackend
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting george>=0.3.1 (from abctest==0.0.1)
  Downloading https://files.pythonhosted.org/packages/27/19/9de575be629e3a41c3ca6b1e2c80e0ae90a2e831436c5f70cc8d72e37ab7/george-0.3.1.tar.gz (1.8MB)
Collecting numpy (from george>=0.3.1->abctest==0.0.1)
  Using cached https://files.pythonhosted.org/packages/63/0c/0261693cc3ad8e2b66e66dc2d2676a2cc17d3efb1c58a70db73754320e47/numpy-1.18.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting scipy (from george>=0.3.1->abctest==0.0.1)
  Downloading https://files.pythonhosted.org/packages/dd/82/c1fe128f3526b128cfd185580ba40d01371c5d299fcf7f77968e22dfcc2e/scipy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (26.1MB)
Collecting pybind11 (from george>=0.3.1->abctest==0.0.1)
  Using cached https://files.pythonhosted.org/packages/4b/4d/ae1c4d8e8b139afa9682054dd42df3b0e3b5c1731287933021b9fd7e9cc4/pybind11-2.4.3-py2.py3-none-any.whl
Building wheels for collected packages: abctest
  Building wheel for abctest (PEP 517): started
  Building wheel for abctest (PEP 517): finished with status 'done'
  Stored in directory: /tmp/pip-ephem-wheel-cache-5ruh8xv_/wheels/e3/5f/a4/652949565a7089970182a50f7eec733873579b601c20f0cb97
Successfully built abctest
Building wheels for collected packages: george
  Building wheel for george (setup.py): started
  Building wheel for george (setup.py): finished with status 'error'
  ERROR: Complete output from command /usr/local/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-bijmj8ct/george/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-i9_120mu --python-tag cp37:
  ERROR: running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/george
  copying george/kernels.py -> build/lib.linux-x86_64-3.7/george
  copying george/utils.py -> build/lib.linux-x86_64-3.7/george
  copying george/gp.py -> build/lib.linux-x86_64-3.7/george
  copying george/modeling.py -> build/lib.linux-x86_64-3.7/george
  copying george/metrics.py -> build/lib.linux-x86_64-3.7/george
  copying george/__init__.py -> build/lib.linux-x86_64-3.7/george
  creating build/lib.linux-x86_64-3.7/george/solvers
  copying george/solvers/basic.py -> build/lib.linux-x86_64-3.7/george/solvers
  copying george/solvers/trivial.py -> build/lib.linux-x86_64-3.7/george/solvers
  copying george/solvers/__init__.py -> build/lib.linux-x86_64-3.7/george/solvers
  copying george/solvers/hodlr.py -> build/lib.linux-x86_64-3.7/george/solvers
  running egg_info
  writing george.egg-info/PKG-INFO
  writing dependency_links to george.egg-info/dependency_links.txt
  writing requirements to george.egg-info/requires.txt
  writing top-level names to george.egg-info/top_level.txt
  reading manifest file 'george.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'george.egg-info/SOURCES.txt'
  copying george/kernel_interface.cpp -> build/lib.linux-x86_64-3.7/george
  creating build/lib.linux-x86_64-3.7/george/include
  copying george/include/.DS_Store -> build/lib.linux-x86_64-3.7/george/include
  creating build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/exceptions.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/george.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/hodlr.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/kernels.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/metrics.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/parser.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/subspace.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/solvers/_hodlr.cpp -> build/lib.linux-x86_64-3.7/george/solvers
  running build_ext
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-bijmj8ct/george/setup.py", line 231, in <module>
      zip_safe=True,
    File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 78, in run
      _build_ext.run(self)
    File "/usr/local/lib/python3.7/distutils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/tmp/pip-install-bijmj8ct/george/setup.py", line 105, in build_extensions
      import numpy
  ModuleNotFoundError: No module named 'numpy'
  ----------------------------------------
  ERROR: Failed building wheel for george
  Running setup.py clean for george
Failed to build george
Installing collected packages: numpy, scipy, pybind11, george, abctest
  Running setup.py install for george: started

Answer 1

Looks to me like the packaging of the george project is somewhat faulty and can not be built cleanly without pre-installing numpy first (as you do in the Dockerfile). Probably if this project were to use the build-system feature of pyproject.toml (PEP 517) and add numpy to its list of requires then it would build without issue.

As an alternative one could consider pre-building their own wheels of the george project and cache/share them for later reuse. In the case of Docker there are good solutions for this.