Calling an MPI dependent Fortran module via Cython

Question

I am trying to call a Fortran library, which uses Open MPI, from Python. I chose to use Cython as the layer between Python and Fortran.

I got a minimal working example running from this answer, but I have problems when it comes to using Open MPI.

Below you find my attempt on including MPI in this example. If I call the module ffunc from a Fortran program, it works, but if I try to access it from Python, I get

mca_base_component_repository_open: unable to open mca_patcher_overwrite: /path/to/mca_patcher_overwrite.so: undefined symbol: mca_patcher_base_patch_t_class (ignored)
mca_base_component_repository_open: unable to open mca_shmem_mmap: /path/to/mca_shmem_mmap.so: undefined symbol: opal_show_help (ignored)
mca_base_component_repository_open: unable to open mca_shmem_posix: /path/to/mca_shmem_posix.so: undefined symbol: opal_shmem_base_framework (ignored)
mca_base_component_repository_open: unable to open mca_shmem_sysv: /path/to/mca_shmem_sysv.so: undefined symbol: opal_show_help (ignored)

As a minimal working example I have following:

The Fortran module to call ffunc.p90:

module ffunc

use mpi

implicit none

integer, parameter :: dp = kind(1.0d0)

contains

subroutine func(a, b, c)
  real(dp), intent(in) :: a
  real(dp), intent(in) :: b
  real(dp), intent(out) :: c
  integer :: ierr

  call mpi_init(ierr)
  c = a + b
  call mpi_finalize(ierr)

end subroutine func


end module ffunc

the Fortran-C wrapper pyffunc.f90:

module fortran_wrapper


use iso_c_binding, only: c_double
use ffunc, only: func

implicit none

contains


subroutine c_func(a, b, c) bind(c)
  real(c_double), intent(in) :: a
  real(c_double), intent(in) :: b
  real(c_double), intent(out) :: c

  call func(a, b, c)

end subroutine c_func


end module fortran_wrapper

the C header pyffunc.h:

extern void c_func(double* a, double* b, double* c);

the Cython wrapper pyffunc.pyx:

cdef extern from 'pyffunc.h':
    void c_func(double* a, double* b, double* c)

def func(double a, double b):
    cdef:
        double c
    c_func(&a, &b, &c)
    print('cython: a = {}, b = {}, c = {}'.format(a, b, c))
    return c

the Python setup.py (and got the includes and libraries from mpif90 showme:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
from numpy import get_include
from os import system

# compile the fortran modules without linking
fortran_mod_comp = 'mpif90 ffunc.f90 -c -o ffunc.o -O3 -fPIC'
print(fortran_mod_comp)
system(fortran_mod_comp)
shared_obj_comp = 'mpif90 pyffunc.f90 -c -o pyffunc.o -O3 -fPIC'
print(shared_obj_comp)
system(shared_obj_comp)

ext_modules = [Extension(# module name:
                         'pyffunc',
                         # source file:
                         ['pyffunc.pyx'],
                         # other compile args for gcc
                         extra_compile_args=[
                             '-fPIC',
                             '-O3',
                             '-I/usr/lib/x86_64-linux-gnu/openmpi/include',
                             '-pthread',
                             '-I/usr/lib/x86_64-linux-gnu/openmpi/lib',
                             ],
                         # other files to link to
                         extra_link_args=['ffunc.o',
                             'pyffunc.o',
                             '-pthread',
                             '-I/usr/lib/x86_64-linux-gnu/openmpi/lib',
                             '-L/usr//lib',
                             '-L/usr/lib/x86_64-linux-gnu/openmpi/lib',
                             '-lmpi_usempif08',
                             '-lmpi_usempi_ignore_tkr',
                             '-lmpi_mpifh',
                             '-lmpi',
                             ])]

setup(name = 'pyffunc',
      cmdclass = {'build_ext': build_ext},
      include_dirs = [get_include()],
ext_modules = ext_modules)

and the Python test.py file:

from pyffunc import func


a = 5
b = 9
c = func(5, 9)

print('python: a = {}, b = {}, c = {}'.format(a, b, c))

Could anyone give me a hint into the right direction? - I searched for all kind of stuff, but I am stuck here.

Answer 1

As suggested by @Gilles Gouaillardet, the trick was indeed to compile Open MPI with the option configure --disable-dlopen. Although I still neither understand the problem, nor the solution. Maybe somebody has a more elaborate answer?