CODEWITHSUNDEEP
pythonmatlabshared-librariespython-multiprocessing
Max Ehrlich
Max Ehrlich

Reputation: 2525

Python Multiprocessing not honoring LD_LIBRARY_PATH for dynamic loading

I am working on a python 3.5 project that uses multiprocessing, the worker process needs to be able to call compiled MATLAB code. In order to not have to set the LD_LIBRARY_PATH environment variable before running python (it causes conflicts with libexpat), I want to have only this worker process using the altered LD_LIBRARY_PATH. I figured this would work fine since processes created with fork() are started with any environment changes made in the parent. I am using 

matlab_runtime_paths = [
    '/usr/local/MATLAB/MATLAB_Runtime/v91/runtime/glnxa64',
    '/usr/local/MATLAB/MATLAB_Runtime/v91/bin/glnxa64',
    '/usr/local/MATLAB/MATLAB_Runtime/v91/sys/os/glnxa64',
    '/usr/local/MATLAB/MATLAB_Runtime/v91/sys/opengl/lib/glnxa64'
]
system_lib_dir = '{}/lib'.format(sys.prefix)

lib_paths = matlab_runtime_paths + [system_lib_dir]
lib_paths_format = ':'.join(lib_paths)
os.putenv('LD_LIBRARY_PATH', lib_paths_format)

to set the environment variable in parent process, then later in the worker process I have 

def matlab_worker(matlab_pipe_end):
    import service
    service.initialize_stub()

which crashes because it is unable to find the library properly. I can see that the environment variable is being set correct, if I add 

def matlab_worker(matlab_pipe_end):
    os.system('echo $LD_LIBRARY_PATH')
    import service
    service.initialize_stub()

the variable is set, I can even do

def matlab_worker(matlab_pipe_end):
    os.system('ldd <path>/service.so')
    import service
    service.initialize_stub()

and see that all shared objects are resolved, yet python fails to honor this setting, crashing on the import. I'm assuming because the dynamic loader doesnt re-read the LD_LIBRARY_PATH variable after a fork (can that be correct? It doesnt sound correct but I cant find any documentation about it)

Is there any way to do what I want or is there something wrong with my code? I should note that this code works fine if the parent process is launched with LD_LIBRARY_PATH=... and that I have made sure there are no conflicting libraries that could be screwing up the process.

Upvotes: 0

Views: 515

Answers (1)

yugr
yugr

Reputation: 21916

I'm assuming because the dynamic loader doesnt re-read the LD_LIBRARY_PATH variable after a fork

That's true, by the time you call putenv dynamic linker has already parsed LD_LIBRARY_PATH and built it's internal tables for resolving symbols. You won't be able to rebuild those without exec.

Alternatively, you can

  • at startup, see if paths have been set and if they haven't, set them and re-exec the script
  • change worker scripts to be standalone so that they can be exec-ed rather than fork-ed
  • start your main app with LD_LIBRARY_PATH set
  • manually load needed libraries via dlopen, using absolute paths

Upvotes: 2

Related Questions