Ignoring "undefined symbol" errors in ctypes library load

Question

I'm loading a not-owned-by-me library with Python's ctypes module as ctypes.CDLL("libthirdparty.so") which produces an error undefined symbol: g_main_context_push_thread_default because libthirdparty.so was overlinking a lot of unneeded / unused stuff like glib.

In this particular case, I can work around this problem successfully by LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 python ...-ing glib, but I'm curious in two generic related questions touching on how ctypes works:

1. Is it possible to ask ctypes / dlopen to ignore undefined symbols, given that I know that these undefined symbols are not used on the code path I'm interested in (or otherwise I would be okay if it crashed in a hard way)?

2. Is it possible to ask ctypes to load several libraries at once or emulate LD_PRELOAD-ing? I tried to do ctypes.CDLL("/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0") in attempt to force-load glib in the Python's process before loading ctypes.CDLL("libthirdparty.so"), but it did not help.

Thanks!

Answer 1

Listing [Python.Docs]: ctypes - A foreign function library for Python.
What you're after, is [Man7]: DLOPEN (3) (emphasis is mine):

RTLD_LAZY

Perform lazy binding. Resolve symbols only as the code that references them is executed. If the symbol is never referenced, then it is never resolved. (Lazy binding is performed only for function references; references to variables are always immediately bound when the shared object is loaded). Since glibc 2.1.1, this flag is overridden by the effect of the LD_BIND_NOW environment variable.

Here's an example:

• common.h:

  #pragma once
  #include <stdio.h>
  
  #define C_TAG "From C"
  #define PRINT_MSG_0() printf("%s - [%s] (%d) - [%s]\n", C_TAG, __FILE__, __LINE__, __FUNCTION__)
  

• dll01.h:

  #pragma once
  #if defined(_WIN32)
  #  define DLL01_EXPORT_API __declspec(dllexport)
  #else
  #  define DLL01_EXPORT_API
  #endif
  
  
  #if defined(__cplusplus)
  extern "C" {
  #endif
  
  DLL01_EXPORT_API int inner();
  
  #if defined(__cplusplus)
  }
  #endif
  

• dll01.c:

  #include "dll01.h"
  #include "common.h"
  
  
  # if !defined(NO_EXPORTS)
  int inner()
  {
      PRINT_MSG_0();
      return 0;
  }
  #endif
  

• dll00.c:

  #include <stdio.h>
  #include "common.h"
  #include "dll01.h"
  
  #if defined(_WIN32)
  #  define DLL00_EXPORT_API __declspec(dllexport)
  #else
  #  define DLL00_EXPORT_API
  #endif
  
  
  #if defined(__cplusplus)
  extern "C" {
  #endif
  
  DLL00_EXPORT_API int simple();
  DLL00_EXPORT_API int innerWrapper();
  
  #if defined(__cplusplus)
  }
  #endif
  
  
  int simple()
  {
      PRINT_MSG_0();
      return 0;
  }
  
  
  int innerWrapper()
  {
      PRINT_MSG_0();
      return inner();
  }
  

• code00.py:

  #!/usr/bin/env python
  
  import ctypes as cts
  import os
  import sys
  
  
  DLL_NAME = "./libdll00.{:s}".format("dll" if sys.platform[:3].lower() == "win" else "so")
  
  
  def main(*argv):
      mode = cts.DEFAULT_MODE
      for arg in argv:
          flag = getattr(os, f"RTLD_{arg.upper()}", None)
          if flag is not None:
              mode |= flag
      print(f"Loading .dll (mode: {mode})...")
      dll = cts.CDLL(DLL_NAME, mode=mode)
      for func_name in ("simple", "innerWrapper"):
          print(f"Loading {func_name} function...")
          func = getattr(dll, func_name)
          func.argtypes = ()
          func.restype = cts.c_int
          res = func()
          print(f"{func_name} returned: {res}")
  
  
  if __name__ == "__main__":
      print(
          "Python {:s} {:03d}bit on {:s}\n".format(
              " ".join(elem.strip() for elem in sys.version.split("\n")),
              64 if sys.maxsize > 0x100000000 else 32,
              sys.platform,
          )
      )
      rc = main(*sys.argv[1:])
      print("\nDone.\n")
      sys.exit(rc)
  
  

Output:

(qaic-env) [cfati@cfati-5510-0:/mnt/e/Work/Dev/StackExchange/StackOverflow/q078879247]> ~/sopr.sh
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###

[064bit prompt]> ls
code00.py  common.h  dll00.c  dll01.c  dll01.h
[064bit prompt]>
[064bit prompt]> gcc -fPIC -shared dll01.c -o libdll01.so
[064bit prompt]> LIBRARY_PATH="${LIBRARY_PATH}:." gcc -fPIC -shared dll00.c -o libdll00.so -ldll01
[064bit prompt]> ls
code00.py  common.h  dll00.c  dll01.c  dll01.h  libdll00.so  libdll01.so
[064bit prompt]> export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:."
[064bit prompt]> ldd libdll00.so
        linux-vdso.so.1 (0x00007ffef55a7000)
        libdll01.so => ./libdll01.so (0x00007f032db60000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f032d94c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f032db6c000)
[064bit prompt]>
[064bit prompt]> # ----- Normal libdll01.so
[064bit prompt]> # --- Symbol resolution: dlopen
[064bit prompt]> python ./code00.py
Python 3.8.10 (default, Jul 29 2024, 17:02:10) [GCC 9.4.0] 064bit on linux

Loading .dll (mode: 0)...
Loading simple function...
From C - [dll00.c] (26) - [simple]
simple returned: 0
Loading innerWrapper function...
From C - [dll00.c] (33) - [innerWrapper]
From C - [dll01.c] (8) - [inner]
innerWrapper returned: 0

Done.

[064bit prompt]> # --- Symbol resolution: lazy
[064bit prompt]> python ./code00.py lazy
Python 3.8.10 (default, Jul 29 2024, 17:02:10) [GCC 9.4.0] 064bit on linux

Loading .dll (mode: 1)...
Loading simple function...
From C - [dll00.c] (26) - [simple]
simple returned: 0
Loading innerWrapper function...
From C - [dll00.c] (33) - [innerWrapper]
From C - [dll01.c] (8) - [inner]
innerWrapper returned: 0

Done.

[064bit prompt]>
[064bit prompt]> gcc -fPIC -shared dll01.c -o libdll01.so -DNO_EXPORTS
[064bit prompt]> # ----- Messed up (undefined "inner" function) libdll01.so
[064bit prompt]> # --- Symbol resolution: dlopen
[064bit prompt]> python ./code00.py
Python 3.8.10 (default, Jul 29 2024, 17:02:10) [GCC 9.4.0] 064bit on linux

Loading .dll (mode: 0)...
Traceback (most recent call last):
  File "./code00.py", line 41, in <module>
    rc = main(*sys.argv[1:])
  File "./code00.py", line 18, in main
    dll = cts.CDLL(DLL_NAME, mode=mode)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ./libdll00.so: undefined symbol: inner
[064bit prompt]>
[064bit prompt]> # --- Symbol resolution: lazy
[064bit prompt]> python ./code00.py lazy
Python 3.8.10 (default, Jul 29 2024, 17:02:10) [GCC 9.4.0] 064bit on linux

Loading .dll (mode: 1)...
Loading simple function...
From C - [dll00.c] (26) - [simple]
simple returned: 0
Loading innerWrapper function...
From C - [dll00.c] (33) - [innerWrapper]
python: symbol lookup error: ./libdll00.so: undefined symbol: inner

Might also worth reading:

• [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer)

• [SO]: Resolving circular shared-object dependencies with ctypes/cffi (@CristiFati's answer)

Regarding your 2^nd question, loading the library beforehand has no effect cause it can be found by the loader anyway, so no matter if loaded manually before or automatically by libthirdparty.so it's still the (same) wrong one.