Python package setup: setup.py with customisation to handle wrapped fortran

Question

I have a python package i would like to distribute. I have the package set-up and am able to download the tarball, unzip and install it using:

python setup.py install

which works fine.

I would also like to upload the package to PyPi, and enable it to be installed using pip.

However, the package contains f2py wrapped fortran, and which needs to be compiled on build with the resulting .so files moved to the eventual installation folder. I am confused as to how to do this using:

python3 setup.py sdist

followed by:

pip3 install pkg_name_here.tar.gz

The reason being that when I run

python3 setup.py sdist

the custom commands are being run, part of which is trying to move the compiled *so files to the installation folder, which has not yet been created. An example of the code outline i have used is in this example here:

from setuptools.command.install import install
from setuptools.command.develop import develop
from setuptools.command.egg_info import egg_info

'''
BEGIN CUSTOM INSTALL COMMANDS
These classes are used to hook into setup.py's install process. Depending on the context:
$ pip install my-package

Can yield `setup.py install`, `setup.py egg_info`, or `setup.py develop`
'''


def custom_command():
    import sys
    if sys.platform in ['darwin', 'linux']:
        os.system('./custom_command.sh')


class CustomInstallCommand(install):
    def run(self):
        install.run(self)
        custom_command()


class CustomDevelopCommand(develop):
    def run(self):
        develop.run(self)
        custom_command()


class CustomEggInfoCommand(egg_info):
    def run(self):
        egg_info.run(self)
        custom_command()

'''
END CUSTOM INSTALL COMMANDS 
'''

setup(
    ...
    cmdclass={
        'install': CustomInstallCommand,
        'develop': CustomDevelopCommand,
        'egg_info': CustomEggInfoCommand,
    },
    ...
)

In my instance the custom_command() compiles and wraps the fortran and copies the lib files to the installation folder.

What I would like to know is if there is a way of only running these custom commands during the installation with pip? i.e avoid custom_command() being run during packaging, and only run during installation.

Update

Following Pierre de Buyl's suggestion i have made some progress, but still do not have this working.

The setup.py file currently looks something like:

 def setup_f90_ext(parent_package='',top_path=''):
    from numpy.distutils.misc_util import Configuration
    from os.path import join

    config = Configuration('',parent_package,top_path)

    tort_src = [join('PackageName/','tort.f90')]
    config.add_library('tort', sources=tort_src,
                          extra_f90_compile_args=['-fopenmp -lgomp -O3'],
                          extra_link_args=['-lgomp'])

    sources = [join('PackageName','f90wrap_tort.f90')]

    config.add_extension(name='',
                          sources=sources,
                          extra_f90_compile_args=['-fopenmp -lgomp -O3'],
                          libraries=['tort'],
                          extra_link_args=['-lgomp'],
                          include_dirs=['build/temp*/'])

    return config

if __name__ == '__main__':

    from numpy.distutils.core import setup
    import subprocess
    import os
    import sys

    version_file = open(os.getcwd()+'/PackageName/'+ 'VERSION')
    __version__ = version_file.read().strip()


    subprocess.call(cmd, shell=True)

    config = {'name':'PackageName',
              'version':__version__,
              'project_description':'Package description',
              'description':'Description',
              'long_description': open('README.txt').read(),#read('README.txt'),
}
    config2 = dict(config,**setup_f90_ext(parent_package='PackageName',top_path='').todict())
    setup(**config2)   

where f90wrap_tort.f90 is the f90wrapped fortran file, and tort.f90 is the original fortran.

This file works with python setup.py install if I run the command twice

The first time I run python setup.py install I get the following error:

    gfortran:f90: ./PackageName/f90wrap_tort.f90
f951: Warning: Nonexistent include directory ‘build/temp*/’ [-Wmissing-include-dirs]
./PackageName/f90wrap_tort.f90:4:8:

     use tort_mod, only: test_node
        1
Fatal Error: Can't open module file ‘tort_mod.mod’ for reading at (1): No such file or directory
compilation terminated.
f951: Warning: Nonexistent include directory ‘build/temp*/’ [-Wmissing-include-dirs]
./PackageName/f90wrap_tort.f90:4:8:

     use tort_mod, only: test_node
        1
Fatal Error: Can't open module file ‘tort_mod.mod’ for reading at (1): No such file or directory

The reason I put the include_dirs=['build/temp*/'] argument in the extension was because I noticed after running python setup.py install the first time tort_mod was being built and stored there.

What I can't figure out is how to get the linking correct so that this is all done in one step.

Can anyone see what I am missing?

Answer 1

After a bit of googling, I suggest the following:

1. Use NumPy's distutils
2. Use the add_library keyword (seen here) for your plain Fortran files. This will build the Fortran files as a library but not try to interface to them with f2py.
3. Pre-build the f90 wrappers with f90wrap, include them in your package archive and specify those files as source in the extension.

I did not test the whole solution as it is a bit time consuming, but this is what SciPy does for some of their modules, see here.

The documentation of NumPy has an item over add_library

EDIT 1: after building with the include_dirs=['build/temp.linux-x86_64-2.7']) config, I obtain this directory structure on the first build attempt.

build/lib.linux-x86_64-2.7
├── crystal_torture
│   ├── cluster.py
│   ├── dist.f90
│   ├── f90wrap_tort.f90
│   ├── graph.py
│   ├── __init__.py
│   ├── minimal_cluster.py
│   ├── node.py
│   ├── node.pyc
│   ├── pymatgen_doping.py
│   ├── pymatgen_interface.py
│   ├── tort.f90
│   ├── tort.py
│   └── tort.pyc
└── crystal_torture.so