CODEWITHSUNDEEP
pythonc++python-2.7numpygdal
gjones
gjones

Reputation: 377

Python C Extension using numpy and gdal giving undefined symbol at runtime

I'm writing a C++ extension for python to speed up a raster image viewer created in-house. I've got working code, but noticed that the speed hadn't increased that much and after profiling a bit deeper realised that it was due to the gdal.ReadAsArray calls, which were python callbacks from the C extension. To get around the overhead of Python C-API when calling python objects I decided I would use the C++ libraries for gdal rather than using the Python callback to the existing gdalDataset. (space isn't a problem). However after implementing the code for this I ran into an error at runtime(the extension compiled fine)

which was

 import getRasterImage_new
 ImportError: /local1/data/scratch/lib/python2.7/site-packages
    /getRasterImage_new.so: undefined symbol: _ZN11GDALDataset14GetRasterYSizeEv

The code below replicates the error(some edits may be needed to run on your machines(ignore the uninitialised variables, it's just whats needed to replicate the error).

Python:

#!/usr/bin/env python
import numpy
from osgeo import gdal
import PythonCTest

print("test starting")
PythonCTest.testFunction(1)
print("test complete")

C++:

#include "Python.h"
#include "numpy/arrayobject.h"
#include "gdal_priv.h"
#include <iostream>

extern "C"{
static PyObject* PythonCTest_testFunction(PyObject* args);
static PyMethodDef PythonCTest_newMethods[] = {
   {"testFunction", (PyCFunction)PythonCTest_testFunction, METH_VARARGS,
      "test function"},
   {NULL,NULL,0,NULL}};

PyMODINIT_FUNC initPythonCTest(void){
   (void)Py_InitModule("PythonCTest",PythonCTest_newMethods);
   import_array();
}
}
GDALDataset* y;
static PyObject* PythonCTest_testFunction(PyObject* args){
   std::cout << "in testFunction\n";
   y->GetRasterYSize();
   std::cout << "doing stuff" << "\n";
   return Py_None;
}

Any suggestions would be very welcome.

EDIT

You can also remove the from osgeo import gdal and the error stull occurs(only just noticed that).

EDIT 2

I forgot to say that I'm compiling my extension using distutils

current setup.py is

#!/usr/bin/env python
from distutils.core import setup, Extension
import os
os.environ["CC"] = "g++"
setup(name='PythonCTest', version='1.0', \
   ext_modules=[Extension('PythonCTest', ['PythonCTest.cpp'],
   extra_compile_args=['--std=c++14','-l/usr/include/gdal', '-I/usr/include/gdal'])])

Upvotes: 2

Views: 574

Answers (1)

user4815162342
user4815162342

Reputation: 155046

A Python extension module is a dynamically loadable (shared) library. When linking a shared library, you need to specify its library dependencies, such as -lgdal and, for that matter, -lpython2.7. Failing to do so results in a library with unresolved symbols, and if those symbols are not provided by the time it is loaded, the loading will fail, as reported by Python.

To resolve the error, you need to add libraries=['gdal'] to Extension constructor. Specifying -lgdal in extra_compile_args won't work because compile arguments, as the name implies, are used for compilation and not for linking.

Note that an unresolved symbol would not go undetected when linking an executable, where the build would fail with a linker error. To get the same diagnostics when linking shared libraries, include -Wl,-zdefs in link arguments.

Upvotes: 2

Related Questions