CODEWITHSUNDEEP
pythonmatlablinkershared-librariesdyld
user1441998
user1441998

Reputation: 459

why can't MATLAB import this python library?

I'd like to use http://www.losonczylab.org/sima/1.2/ from within MATLAB.

I can run it fine from python.

I can import example dependencies in MATLAB.

$ /opt/local/bin/python 
Python 2.7.9 (default, Dec 13 2014, 15:13:49) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin 
Type "help", "copyright", "credits" or "license" for more information. 
>>> from sima import sequence 
>>> sequence 
<module 'sima.sequence' from '/Users/eflister/Library/Python/2.7/lib/python/site-packages/sima/sequence.pyc'> 
>>> from sima import ImagingDataset 
>>> ImagingDataset 
<class 'sima.imaging.ImagingDataset'>

in MATLAB: 

>> pyversion /opt/local/bin/python 
>> pyversion 

version: '2.7' 
executable: '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Cont...' 
library: '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib' 
home: '/opt/local/Library/Frameworks/Python.framework/Versions/2.7' 
isloaded: 0 

>> import py.picos.tools.available_solvers 
>> available_solvers 

ans = 

Python list with no properties. 

['cvxopt', 'mosek7'] 

>> import py.numpy.empty 
>> empty(0) 

ans = 

Python ndarray with properties: 

T: [1x1 py.numpy.ndarray] 
base: [1x1 py.NoneType] 
ctypes: [1x1 py.numpy.core._internal._ctypes] 
data: [1x0 py.buffer] 
dtype: [1x1 py.numpy.dtype] 
flags: [1x1 py.numpy.flagsobj] 
flat: [1x1 py.numpy.flatiter] 
imag: [1x1 py.numpy.ndarray] 
itemsize: 8 
nbytes: 0 
ndim: 1 
real: [1x1 py.numpy.ndarray] 
shape: [1x1 py.tuple] 
size: 0 
strides: [1x1 py.tuple] 

[] 

>> import py.sima.sequence 
Error using import 
Import argument 'py.sima.sequence' cannot be found or cannot be imported. 

>> import py.sima.Sequence 
Error using import 
Import argument 'py.sima.Sequence' cannot be found or cannot be imported. 

>> import py.sima.Sequence.create 
Error using import 
Import argument 'py.sima.Sequence.create' cannot be found or cannot be imported. 

>> import py.sima.sequence.create 
Error using import 
Import argument 'py.sima.sequence.create' cannot be found or cannot be imported. 

>> import py.sima.sequence 
Error using import 
Import argument 'py.sima.sequence' cannot be found or cannot be imported. 

>> import py.sima.Sequence 
Error using import 
Import argument 'py.sima.Sequence' cannot be found or cannot be imported. 

>> import py.sima.ImagingDataset 
Error using import 
Import argument 'py.sima.ImagingDataset' cannot be found or cannot be imported. 

>> import py.sima.imagingdataset 
Error using import 
Import argument 'py.sima.imagingdataset' cannot be found or cannot be imported. 

>> import py.sima 
Error using import 
Import argument 'py.sima' cannot be found or cannot be imported.

How can I figure out why sima can't be imported?

sima is on the path:

>> pp = py.pprint.PrettyPrinter

pp = 

  Python PrettyPrinter with no properties.

    <pprint.PrettyPrinter instance at 0x129425bd8>

>> pp.pprint(py.sys.path)
['',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/readline',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/Users/eflister/Library/Python/2.7/lib/python/site-packages',
 '/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages']

I came up with the following tedious method to find the problem imports -- is there a better way?

  1. define a module as in http://www.mathworks.com/help/matlab/matlab_external/call-user-defined-custom-module.html
  2. copy in imports from source files in library that MATLAB won't load
  3. comment out suspect imports until MATLAB call into module succeeds
  4. rename the module and repeat (once successfully loaded, MATLAB has to restart to see new changes)

I only had to do this 29 times to isolate the following problematic imports in sima. :)

from PIL import Image

from skimage.measure import approximate_polygon
from skimage.measure import find_contours
from skimage.filter import threshold_otsu
from skimage import transform as tf

from sklearn.decomposition import FastICA

import scipy.io
from scipy.stats import uniform
from scipy.linalg import inv
from scipy.sparse.linalg import eigsh, eigs
from scipy.sparse.linalg import inv
from scipy import ndimage
from scipy.ndimage import measurements

import cv2

# note the following are OK:
from scipy.linalg import eig, eigh, norm
from scipy import sparse
from multiprocessing import Pool
import multiprocessing.util as util

running this code in python shows no import exceptions:

lines = """
import h5py
from PIL import Image

from skimage.measure import approximate_polygon
from skimage.measure import find_contours
from skimage.filter import threshold_otsu
from skimage import transform as tf

from sklearn.decomposition import FastICA

import scipy.io
from scipy.stats import uniform
from scipy.linalg import inv
from scipy.sparse.linalg import eigsh, eigs
from scipy.sparse.linalg import inv
from scipy import ndimage
from scipy.ndimage import measurements

import cv2
"""

def tryImport(i):
    try:
        exec(i)
    except ImportError as e:
        print i
        print e
        print

map(tryImport, filter(lambda x: len(x) > 0, lines.splitlines()))

but in MATLAB, it gives the output:

import h5py
cannot import name _errors

from PIL import Image
dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PIL/_imaging.so, 2): Library not loaded: /opt/local/lib/libtiff.5.dylib
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PIL/_imaging.so
  Reason: Incompatible library version: _imaging.so requires version 8.0.0 or later, but libtiff.5.dylib provides version 6.0.0

from skimage.measure import approximate_polygon
cannot import name _ni_support

from skimage.measure import find_contours
cannot import name _ni_support

from skimage.filter import threshold_otsu
cannot import name _ni_support

from skimage import transform as tf
cannot import name hough_ellipse

from sklearn.decomposition import FastICA
dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/optimize/_lbfgsb.so, 2): Symbol not found: __gfortran_transfer_character_write
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/optimize/_lbfgsb.so
  Expected in: /Applications/MATLAB_R2015a.app/sys/os/maci64/libgfortran.3.dylib
 in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/optimize/_lbfgsb.so

import scipy.io
cannot import name doccer

from scipy.stats import uniform
dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/special/_ufuncs.so, 2): Symbol not found: __gfortran_stop_numeric_f08
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/special/_ufuncs.so
  Expected in: /Applications/MATLAB_R2015a.app/sys/os/maci64/libgfortran.3.dylib
 in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/special/_ufuncs.so

from scipy.sparse.linalg import eigsh, eigs
dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/_arpack.so, 2): Symbol not found: __gfortran_transfer_array_write
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/_arpack.so
  Expected in: /Applications/MATLAB_R2015a.app/sys/os/maci64/libgfortran.3.dylib
 in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/_arpack.so

from scipy.sparse.linalg import inv
dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/_arpack.so, 2): Symbol not found: __gfortran_transfer_array_write
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/_arpack.so
  Expected in: /Applications/MATLAB_R2015a.app/sys/os/maci64/libgfortran.3.dylib
 in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/eigen/arpack/_arpack.so

from scipy import ndimage
cannot import name _ni_support

from scipy.ndimage import measurements
cannot import name _ni_support

import cv2
dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/cv2.so, 2): Library not loaded: /opt/local/lib/libtiff.5.dylib
  Referenced from: /opt/local/lib/libopencv_highgui.2.4.dylib
  Reason: Incompatible library version: libopencv_highgui.2.4.dylib requires version 8.0.0 or later, but libtiff.5.dylib provides version 6.0.0

i found that this method shows the errors from matlab:

>> py.importlib.import_module('sima')
Python Error: cannot import name hough_ellipse

so is there any hope in proceeding from here?

I don't really understand the errors, as best I can surmise, some of them are:

matlab's libtiff.5.dylib provides version 6.0.0, but PIL and opencv require >= 8.0.0

matlab's libgfortran.3.dylib doesn't work for scipy's _lbfgsb.so, _ufuncs.so, _arpack.so

would there be some way to compile scipy/PIL/opencv against MATLAB's libraries?

would it be extremely painful? :)

i've found the following 3rd party python-from-matlab bridges:

are there others to consider? opinions/experience on which is most promising for this scenario? i'm guessing matpy...

pymex was no good

matpy is no good:

>> py_shell
py> import sima
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/eflister/Library/Python/2.7/lib/python/site-packages/sima/__init__.py", line 6, in <module>
    from sima.imaging import ImagingDataset
  File "/Users/eflister/Library/Python/2.7/lib/python/site-packages/sima/imaging.py", line 26, in <module>
    import h5py
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/h5py/__init__.py", line 23, in <module>
    from . import _conv
  File "h5py/h5t.pxd", line 14, in init h5py._conv (/opt/local/var/macports/build/_opt_mports_dports_python_py-h5py/py27-h5py/work/h5py-2.5.0/h5py/_conv.c:6958)
  File "h5py/h5t.pyx", line 139, in init h5py.h5t (/opt/local/var/macports/build/_opt_mports_dports_python_py-h5py/py27-h5py/work/h5py-2.5.0/h5py/h5t.c:20282)
  File "h5py/h5t.pyx", line 73, in h5py.h5t.lockid (/opt/local/var/macports/build/_opt_mports_dports_python_py-h5py/py27-h5py/work/h5py-2.5.0/h5py/h5t.c:2562)
  File "h5py/h5t.pyx", line 42, in h5py.h5t.typewrap (/opt/local/var/macports/build/_opt_mports_dports_python_py-h5py/py27-h5py/work/h5py-2.5.0/h5py/h5t.c:2196)
RuntimeError: Interface initialization failed (Not a datatype object)
py> import sima
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/eflister/Library/Python/2.7/lib/python/site-packages/sima/__init__.py", line 6, in <module>
    from sima.imaging import ImagingDataset
  File "/Users/eflister/Library/Python/2.7/lib/python/site-packages/sima/imaging.py", line 33, in <module>
    import sima.misc
  File "/Users/eflister/Library/Python/2.7/lib/python/site-packages/sima/misc/__init__.py", line 21, in <module>
    from skimage import transform as tf
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/skimage/transform/__init__.py", line 1, in <module>
    from ._hough_transform import (hough_ellipse, hough_line,
  File "skimage/transform/_hough_transform.pyx", line 13, in init skimage.transform._hough_transform (skimage/transform/_hough_transform.c:22321)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/skimage/draw/__init__.py", line 2, in <module>
    from .draw3d import ellipsoid, ellipsoid_stats
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/skimage/draw/draw3d.py", line 3, in <module>
    from scipy.special import (ellipkinc as ellip_F, ellipeinc as ellip_E)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/special/__init__.py", line 586, in <module>
    from ._ufuncs import *
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/special/_ufuncs.so, 2): Symbol not found: __gfortran_stop_numeric_f08
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/special/_ufuncs.so
  Expected in: /Applications/MATLAB_R2015a.app/sys/os/maci64/libgfortran.3.dylib
 in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/special/_ufuncs.so
py> import sima
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/eflister/Library/Python/2.7/lib/python/site-packages/sima/__init__.py", line 6, in <module>
    from sima.imaging import ImagingDataset
  File "/Users/eflister/Library/Python/2.7/lib/python/site-packages/sima/imaging.py", line 33, in <module>
    import sima.misc
  File "/Users/eflister/Library/Python/2.7/lib/python/site-packages/sima/misc/__init__.py", line 21, in <module>
    from skimage import transform as tf
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/skimage/transform/__init__.py", line 1, in <module>
    from ._hough_transform import (hough_ellipse, hough_line,
ImportError: cannot import name hough_ellipse
py> import sima
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/eflister/Library/Python/2.7/lib/python/site-packages/sima/__init__.py", line 6, in <module>
    from sima.imaging import ImagingDataset
  File "/Users/eflister/Library/Python/2.7/lib/python/site-packages/sima/imaging.py", line 33, in <module>
    import sima.misc
  File "/Users/eflister/Library/Python/2.7/lib/python/site-packages/sima/misc/__init__.py", line 21, in <module>
    from skimage import transform as tf
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/skimage/transform/__init__.py", line 1, in <module>
    from ._hough_transform import (hough_ellipse, hough_line,
ImportError: cannot import name hough_ellipse
py> import picos.tools
py> print picos.tools.available_solvers()
['cvxopt', 'mosek7']

i'm starting to think wormholing to a separate process sounds good...

but what about using LD_PRELOAD to start matlab with updated libgfortran and libtiff?

the versions i want are located:

/opt/local/lib/libgcc/libgfortran.3.dylib 
/opt/local/lib/libtiff.5.dylib

from within a fresh matlab session, the following didn't help.

>> setenv('DYLD_LIBRARY_PATH',['/opt/local/lib:' getenv('DYLD_LIBRARY_PATH')])
>> getenv('DYLD_LIBRARY_PATH')

ans =

/opt/local/lib:/Applications/MATLAB_R2015a.app/sys/os/maci64:/Applications/MATLAB_R2015a.app/bin/maci64/../../Contents/MacOS:/Applications/MATLAB_R2015a.app/bin/maci64:/Applications/MATLAB_R2015a.app/extern/lib/maci64:/Applications/MATLAB_R2015a.app/runtime/maci64:/Applications/MATLAB_R2015a.app/sys/java/jre/maci64/jre/lib/./native_threads:/Applications/MATLAB_R2015a.app/sys/java/jre/maci64/jre/lib/./server:/Applications/MATLAB_R2015a.app/sys/java/jre/maci64/jre/lib/./lib/jli

but!!!

~$ DYLD_INSERT_LIBRARIES=/opt/local/lib/libgcc/libgfortran.3.dylib:/opt/local/lib/libtiff.5.dylib /Applications/MATLAB_R2015a.app/bin/matlab

solves all the import problems!!!

>> import py.sima
Error using import
Import argument 'py.sima' cannot be found or cannot be imported.

>> import py.sima.Sequence % succeeds!
>> import py.sima.ImagingDataset % succeeds!
>> x = py.sima.ImagingDataset([1,2],'~')
Python Error: ImagingDataset objects must be initialized with a list of sequences.

>> x = py.numpy.random.random([2,3,4])

x = 

  Python ndarray with properties:

           T: [1x1 py.numpy.ndarray]
        base: [1x1 py.NoneType]
      ctypes: [1x1 py.numpy.core._internal._ctypes]
        data: [1x192 py.buffer]
       dtype: [1x1 py.numpy.dtype]
       flags: [1x1 py.numpy.flagsobj]
        flat: [1x1 py.numpy.flatiter]
        imag: [1x1 py.numpy.ndarray]
    itemsize: 8
      nbytes: 192
        ndim: 3
        real: [1x1 py.numpy.ndarray]
       shape: [1x3 py.tuple]
        size: 24
     strides: [1x3 py.tuple]

    [[[ 0.4158531   0.18912596  0.20743597  0.92631168]
      [ 0.0371785   0.48093953  0.88221637  0.85687334]
      [ 0.2267605   0.79464779  0.97438662  0.47791076]]

     [[ 0.05602249  0.06596678  0.05691491  0.02706325]
      [ 0.28477145  0.74781928  0.56700236  0.14500885]
      [ 0.87999554  0.78932748  0.24473039  0.18383564]]]

>> x = py.sima.Sequence.create('ndarray',x)
The class "py.sima.Sequence" is undefined.
Perhaps Java is not running.

>> py.sima.segment.STICA()

ans = 

  Python STICA with no properties.

    <sima.segment.stica.STICA object at 0x138e15890>

>> py.sima.segment.PlaneCA1PC()

ans = 

  Python PlaneCA1PC with no properties.

    <sima.segment.ca1pc.PlaneCA1PC object at 0x138e158d0>

we are finally SO CLOSE. but why can't i get sima.Sequence.create() to run!??!

Upvotes: 3

Views: 4871

Answers (2)

user1441998
user1441998

Reputation: 459

In combination with the DYLD_INSERT_LIBRARIES, we can't touch a sima.Sequence directly in MATLAB because it has attributes with leading underscores, which MATLAB doesn't allow as noted by @user3666197.

but we can call python functions that use sima.Sequence no problem:

https://github.com/losonczylab/sima/issues/183#issuecomment-109097816

finally, sweet success!

addendum:
on Windows, the problem is not in libgfortran/libtiff, but with shapely's use of msvcr.

Windows python uses msvcr90.dll, but MATLAB uses msvcr100.dll.

in shapely,
replacing
CDLL(find_library('c'))
with
CDLL('C:\\Program Files\\MATLAB\\R2015a\\bin\\win64\\msvcr100.dll')
makes it all work.
this answer seems to be a method that would determine that path automatically, appropriate for wherever you are invoking shapely from.

Upvotes: 0

user3666197
user3666197

Reputation: 1

A: MATLAB has set of explicit limits to which it does support / NOT support in python code & classes

While your both wish & will to get it work together, is recognised, MATLAB does not support python without limits.

MATLAB has explicitly listed features, it does not support. One of them is a support for object-names starting with an underscore character.

Many of this very error messages, you have posted, confirm this fact.

More on UN-SUPPORTED features can be checked in >>> Limitations to Python Support

Going through the list shows, how immense would be the budget to pay in case you would like to re-engineer / re-design all the packages, that are non-conform with these UN-SUPPORTED features.

What to do with this?

If you are indeed heavily dependent on this dual-world MATLAB/python, you might become interested into another architecture of how to make things work.

Try to integrate a process-to-process messaging via ZeroMQ et al, and start using a distributed processing, using native ( working ) modules on each side.

Would be much faster than trial/error on imports, that would never fly.

Upvotes: 3

Related Questions