Python/Pip packaging; how to move built files to install directory

Question

I have been working on a Python package which wraps some C++ libraries that need to be built from source. I build these with CMake, and I want the whole thing to be 'pip install'able in the end. I am almost there, however I am having problems getting the libraries built by CMake to end up in the final Python installation directory.

I managed to get them into the final 'wheel', oddly enough, but they aren't in my site_packages directory.

My setup.py file looks like this:

import os
import re
import sys
import sysconfig
import site
import platform
import subprocess
import pathlib

from distutils.version import LooseVersion
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext as build_ext_orig

class CMakeExtension(Extension):
    def __init__(self, name, sourcedir=''):
        Extension.__init__(self, name, sources=[])
        self.sourcedir = os.path.abspath(sourcedir)

class CMakeBuild(build_ext_orig):
    def run(self):
        try:
            out = subprocess.check_output(['cmake', '--version'])
        except OSError:
            raise RuntimeError("CMake must be installed to build the following extensions: " +
                               ", ".join(e.name for e in self.extensions))

        if platform.system() == "Windows":
            raise RuntimeError("Sorry, pyScannerBit doesn't work on Windows platforms. Please use Linux or OSX.")

        for ext in self.extensions:
            self.build_extension(ext)

    def build_extension(self, ext):
        extdir = os.path.abspath(os.path.dirname(self.get_ext_fullpath(ext.name)))
        cmake_args = ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=' + extdir,
                      '-DPYTHON_EXECUTABLE=' + sys.executable,
                      '-DCMAKE_VERBOSE_MAKEFILE:BOOL=OFF',
                      '-Wno-dev',
                      '-DCMAKE_RUNTIME_OUTPUT_DIRECTORY=' + extdir,
                      '-DSCANNERBIT_STANDALONE=True',
                      '-DCMAKE_INSTALL_RPATH=$ORIGIN',
                      '-DCMAKE_BUILD_WITH_INSTALL_RPATH:BOOL=ON',
                      '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON',
                      '-DCMAKE_INSTALL_PREFIX:PATH=' + extdir,
                     ]

        cfg = 'Debug' if self.debug else 'Release'
        build_args = ['--config', cfg]

        if platform.system() == "Windows":
            cmake_args += ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_{}={}'.format(cfg.upper(), extdir)]
            if sys.maxsize > 2**32:
                cmake_args += ['-A', 'x64']
            build_args += ['--', '/m']
        else:
            cmake_args += ['-DCMAKE_BUILD_TYPE=' + cfg]
            build_args += ['--', '-j2']

        env = os.environ.copy()
        env['CXXFLAGS'] = '{} -DVERSION_INFO=\"{}\"'.format(env.get('CXXFLAGS', ''),
                                                              self.distribution.get_version())

        if not os.path.exists(self.build_temp):
            os.makedirs(self.build_temp)

        # untar ScannerBit tarball
        subprocess.check_call(['tar','-C','pyscannerbit/scannerbit/untar/ScannerBit','-xf','pyscannerbit/scannerbit/ScannerBit_stripped.tar','--strip-components=1'], cwd=ext.sourcedir, env=env)

        # First cmake
        subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
        # Build all the scanners
        subprocess.check_call(['cmake', '--build', '.', '--target', 'multinest'] + build_args, cwd=self.build_temp)
        # Re-run cmake to detect built scanner plugins
        subprocess.check_call(['cmake', ext.sourcedir], cwd=self.build_temp)
        # Main build
        subprocess.check_call(['cmake', '--build', '.'] + build_args, cwd=self.build_temp)
        # Install
        #subprocess.check_call(['cmake', '--build', '.', '--target', 'install'], cwd=self.build_temp)

setup(
    name='pyscannerbit',
    version='0.0.8',
    author='Ben Farmer',
    # Add yourself if you contribute to this package
    author_email='ben.farmer@gmail.com',
    description='A python interface to the GAMBIT scanning module, ScannerBit',
    long_description='',
    ext_modules=[CMakeExtension('_interface')],
    cmdclass=dict(build_ext=CMakeBuild),
    zip_safe=False,
    packages=['pyscannerbit'],
)

As you can see, I am telling CMake to build the libraries in 'extdir', which it turns out is

/tmp/pip-req-build-d7mfvn1a/build/lib.linux-x86_64-3.6

I had assumed that the files would just be copied from here (or some other temporary directory?) into the final install path in bulk, but perhaps it doesn't work like that (though as I said earlier, these built files do end up in the wheel that is generated). Do these built files need to be added to MANIFEST.in or some 'package_data' entry or something like that? Currently they are not listed anywhere like that, since it was my understanding that those were for moving files around pre-build, not post-build. Currently I only use MANIFEST.in to make sure my sdist tarball gets filled correctly.

For completeness, I am building the package with pip as follows:

python setup.py sdist
pip install -v dist/pyscannerbit-0.0.8.tar.gz

This is just so I know that the build from the tarball works, for later use with PyPI.

The source is on github if you want to try it out: https://github.com/bjfar/pyscannerbit

Python/Pip packaging; how to move built files to install directory

Answers (1)

Related Questions