Embedding Python which uses Numpy in C++ doesn't work in library dynamically loaded at runtime

Question

I want to embed Python in C++ in Ubuntu Linux via a dynamically loaded library (dlopen/dlsym). When I embed a Python module which imports numpy via a single C++ statically linked executable everything works fine. When I use a dynamically loaded library without trying to import numpy everything works fine. However, when I try to import numpy from within a library which was dynamically loaded it fails.

---

To start, here's a simple example without dynamic libraries which works

test_static.cpp

#include <iostream>
#include <Python.h>

int main(){

  Py_Initialize();

  PyObject* pName = PyUnicode_FromString("mymodule");
  PyObject* pModule = PyImport_Import(pName);
  if (!pModule){
    std::cout << "PyImport_Import() failed for " << "test" << std::endl;
    PyErr_Print();
    exit(-1);
  }

  PyObject* pDict = PyModule_GetDict(pModule);
  if (!pDict){
    std::cout << "PyModule_GetDict() failed" << std::endl;
    exit(-1);
  }
  PyObject* pClass = PyDict_GetItemString(pDict,"myclass");

  if (!PyCallable_Check(pClass)) {
    std::cout << "PyCallable_Check(pClass) failed" << std::endl;
    exit(-1);
  }
  PyObject* pInstance = PyObject_CallObject(pClass, NULL);

  pName = PyUnicode_FromString("mymethod");
  PyObject* result=PyObject_CallMethodObjArgs(pInstance, pName,NULL);
  if (!result){
    std::cout << "Error calling PyObject_CallMethodObjArgs" << std::endl;
    PyErr_Print();
    exit(-1);
  }
  Py_DECREF(pName);
  Py_DECREF(pModule);
  Py_Finalize();

  return 0;
}

This is built with

g++ -g -O0 -o test_embed_python_static test_static.cpp -I/usr/include/python3.7m -lpython3.7m

My Python module is:

mymodule.py

import numpy as np

class myclass:
  def __init__(self):
    self.something="asdf"
    self.a=np.zeros( (5),dtype=float)
  def mymethod(self):
    print("Called myclass.mymethod()")
    print('%d'%(self.a.size))

This Python module lives in the local directory and I set the environment variable export PYTHONPATH=.

The output is:

user@host:~/test_embed_python$ ./test_embed_python_static
Called myclass.mymethod()
5

---

Here's the example which uses dynamic libraries:

test_dynamic.cpp

#include <dlfcn.h>
typedef void MyCppFunctionHandle();
int main(){
  void* soHandle = dlopen("/home/user/lib/libtestlib.so", RTLD_NOW );
  MyCppFunctionHandle* functionHandle=(MyCppFunctionHandle*)dlsym(soHandle,"MyCppFunction");
  functionHandle();
  return 0;
}

test_lib.cpp

#include <iostream>
#include <Python.h>

extern "C"
void MyCppFunction(){

  Py_Initialize();

  PyObject* pName = PyUnicode_FromString("mymodule");
  PyObject* pModule = PyImport_Import(pName);
  if (!pModule){
    std::cout << "PyImport_Import() failed for " << "mymodule" << std::endl;
    PyErr_Print();
    exit(-1);
  }

  PyObject* pDict = PyModule_GetDict(pModule);
  if (!pDict){
    std::cout << "PyModule_GetDict() failed" << std::endl;
    exit(-1);
  }

  PyObject* pClass = PyDict_GetItemString(pDict,"myclass");

  if (!PyCallable_Check(pClass)) {
    std::cout << "PyCallable_Check(pClass) failed" << std::endl;
    exit(-1);
  }
  PyObject* pInstance = PyObject_CallObject(pClass, NULL);

  pName = PyUnicode_FromString("mymethod");
  PyObject* result=PyObject_CallMethodObjArgs(pInstance, pName,NULL);
  if (!result){
    std::cout << "Error calling PyObject_CallMethodObjArgs" << std::endl;
    PyErr_Print();
    exit(-1);
  }
  Py_DECREF(pName);
  Py_DECREF(pModule);
  Py_Finalize();
}

Build library and main executable:

g++ -g -O0 -fPIC -shared -o libtestlib.so test_lib.cpp -I/usr/include/python3.7m -lpython3.7m
mv libtestlib.so ~/lib/
g++ -g -O0 -o test_embed_python_dynamic test_dynamic.cpp -ldl

Using the same mymodule.py as above and PYTHONPATH set to . again.

Output:

user@host:~/test_embed_python$ ./test_embed_python_dynamic 
Dynamic library opened successfully
Function loaded successfully
Calling function
PyImport_Import() failed for mymodule
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 40, in <module>
    from . import multiarray
  File "/usr/lib/python3/dist-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/usr/lib/python3/dist-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: /usr/lib/python3/dist-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyExc_ImportError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/code/test/test_embed_python/mymodule.py", line 1, in <module>
    import numpy as np
  File "/usr/lib/python3/dist-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 71, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
  your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
  1. Check that you are using the Python you expect (you're using /usr/bin/python3),
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy versions you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

     Note: this error has many possible causes, so please don't comment on
     an existing issue about this - open a new one instead.

Original error was: /usr/lib/python3/dist-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyExc_ImportError

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ImportError: /usr/lib/python3/dist-packages/apt_pkg.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyExc_ValueError

Original exception was:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 40, in <module>
    from . import multiarray
  File "/usr/lib/python3/dist-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/usr/lib/python3/dist-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: /usr/lib/python3/dist-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyExc_ImportError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/test_embed_python/mymodule.py", line 1, in <module>
    import numpy as np
  File "/usr/lib/python3/dist-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 71, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
  your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
  1. Check that you are using the Python you expect (you're using /usr/bin/python3),
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy versions you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log


     Note: this error has many possible causes, so please don't comment on
     an existing issue about this - open a new one instead.

Original error was: /usr/lib/python3/dist-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so: undefined symbol: PyExc_ImportError

---

This question on the Numpy Github looks similar but the actual solution was never posted:

https://github.com/numpy/numpy/issues/14480

Answer 1

I stumbled on the solution here. It's still unclear to me what the problem is and why this fixes it, but adding the line

void*const libpython_handle = dlopen("libpython3.7m.so", RTLD_LAZY | RTLD_GLOBAL);

before Py_Initialize(); in my test_lib.cpp fixes the problem.