CODEWITHSUNDEEP
pythondllpathdistutils
Marie R
Marie R

Reputation: 299

Python extension modules on Windows - controlling the DLL path for dependencies

I am porting a swig extension module that was designed for Python 2 to Python 3 under Windows 10 with a stock Python 3.9.1. The module builds properly and depends on several Windows dynamic load libraries (*.dll) that I have packaged into the data directory.

Importing the module fails, and using gflags to track down the problem shows that the dependent dll's are not being loaded as the search path does not include the module name: lib/site-packages/dbxml. When I manually move the dll's into the lib/site-packages, they can be located and successfully loaded.

I am confused on how to modify setup to generate a DLL path that includes the module name. I saw this post that seemed to imply that I could simply use the data_files option, but that does not appear to work in my case.

Here is what I am doing with what I believe to be the relevant variables replaced with their values:

  setup(name = "dbxml",
    version = "6.1.4",
    description = "Berkeley DB XML Python API",
    long_description = """removed...""",
    author = "Oracle",
    author_email = "[email protected]",
    url = "http://www.oracle.com",
    py_modules = ["dbxml"],
    ext_modules = [Extension("_dbxml", ["dbxml_python3_wrap.cpp"],
                             include_dirs = INCLUDES,
                             library_dirs = ['../../../lib', '../../build_windows/Release', 
                                             '../../../db-6.2.23/build_windows/Release', 
                                             '../../../xqilla/lib', 
                                             '../../../xerces-c-src/Build/Win32/VC10'],
                             define_macros = DEFINES,
                             libraries = ['libdbxml61', 'libdb62', 'xqilla23', 'xerces-c_3'],
                             extra_compile_args = ['/GR', '/EHsc']
                             )],
    # The DLLs below are copied into lib/site-packages/dbxml
    # but the DLL search path is:
    # C:\Users\[...omitted...]\python3.9\lib\site-packages;
    # C:\Users\[...omitted...]\python3.9;
    # C:\WINDOWS\SYSTEM32
    # and they are not found.
    data_files = [('lib/site-packages/dbxml',
                   ['../../../bin/libdbxml61.dll',
                    '../../../bin/libdb62.dll', 
                    '../../../bin/xqilla23.dll', 
                    '../../../bin/xerces-c_3_1.dll', 
                    '../../build_windows/zlib1.dll', 
                    '../../build_windows/zlibwapi.dll'
                   ])
                 ]
   )

My understanding is that Extension's runtime_library_dirs keyword argument only controls the runtime library path on UNIX-like operating systems and is ignored on Windows 10. Obviously, I am doing something wrong. Any suggestions on how to fix this would be appreciated.

Thank you - Marie

Upvotes: 0

Views: 1123

Answers (1)

Marie R
Marie R

Reputation: 299

It appears that the DLL library loading behavior changed in Python 3.8. The path for DLLs in Windows must be set explicitly using os module function add_dll_directory. I was able to resolve this by adding:

# add peer directory to dll search path
import os
dll_dir = os.path.join(os.path.dirname(__file__), "dbxml")
os.add_dll_directory(dll_dir)

There may be a better way to do this, but this will let the extension module load dyanmicaly linked libraries from a peer directory in site-packages.

Upvotes: 2

Related Questions