Errors using Sympy's `lambdify` with `IPython.parallel`

Question

For a numerical computation I am working on, I need to define a large Liouvillian matrix. Rather than program it in element by element, which is tedious and error-prone, I have used Sympy to construct it algebraically, and then use lambdify to make numpy matrices for use in numerical work. This works fine for small tasks, but weird errors happen when I distribute these functions to worker engines using IPython.parallel.

Here, for example, I construct a dumb sympy matrix that doesn't mean anything:

import sympy as s
from sympy.abc import x,y
s.init_printing()

element = lambda n, m : m * x**n if (n+m) % 3 else y
L = s.Matrix([[element(n,m) for m in range(9)] for n in range(9)])

In the case of this example, I could have constructed a numpy matrix directly using the same nested loop, but this isn't true of the matrix in my actual problem. Anyway it is nice to see it written out in algebraic notation before plugging in numbers.

I use lambdify to get a Numpy matrix for my numerical work:

numer_L = s.lambdify((x,y), L, 'numpy')
numer_L(3,4)  # gives numpy matrix for x=3, y=4

Say I want to do a computation involving this matrix (determinant, say) evaluated at multiple values of y:

# in series
import numpy
s_result = list(map(lambda y: numpy.linalg.det(numer_L(3,y)), range(30)))

This example isn't expensive, but if it were, I would distribute the task like:

# set up parallel environment. 2 engines started with `ipcluster start -n 2`
from IPython.parallel import Client
rc = Client()
dview = rc[:]

# in parallel
# do imports and push our lambda function over
dview.execute('import numpy')
dview.push(dict(numer_L=numer_L))
p_result = dview.map_sync(lambda y: numpy.linalg.det(numer_L(3,y)), range(30))

I get the following error:

[0:apply]: 

---------------------------------------------------------------------------

NameError                                 Traceback (most recent call last)<string> in <module>()

<ipython-input-5-1f431230550c> in <lambda>(y)

/Users/tkb/.virtualenvs/sympy/lib/python2.7/site-packages/numpy/__init__.pyc in <lambda>(x, y)

NameError: global name 'ImmutableMatrix' is not defined



[1:apply]: 

---------------------------------------------------------------------------

NameError                                 Traceback (most recent call last)<string> in <module>()

<ipython-input-5-1f431230550c> in <lambda>(y)

/Users/tkb/.virtualenvs/sympy/lib/python2.7/site-packages/numpy/__init__.pyc in <lambda>(x, y)

NameError: global name 'ImmutableMatrix' is not defined

That didn't work because apparently the lambda function needs ImmutableMatrix defined, which I hadn't ever heard of, and is not even the type of the matrix we lambdified:

type(L)  # sympy.matrices.dense.MutableDenseMatrix

In any case, I don't want my engines running any Sympy code. The task I want to distribute is numerical, not algebraic, and hopefully lambdify has generated numpy code that can run on its own.

What is the proper way to generate parallelizable numpy code from sympy?

Versions

This was done with Python 2.7.3, IPython 1.1.0, Sympy 0.7.4.1, and Numpy 1.8.0. The notebook I used to write this question is accessible on nbviewer.

Answer 1

You can add things to the namespace of lambdify with the second argument, like

>>> lambdify(x, Matrix([[x, 2], [3, 4]]), [{'ImmutableMatrix': numpy.matrix}, "numpy"])(1)
matrix([[1, 2],
   [3, 4]])

But this should be unnecessary with the latest version of SymPy, as that mapping is already done with "numpy", as you can see here. Basically all lambdify does is create a lambda string and a namespace of name translations, and then execs the string in that namespace. I suspect the issue may lie somewhere with IPython parallel, or your use of it. One suggestion, with your

dview.execute('import numpy')
dview.execute('from numpy import matrix as ImmutableMatrix')
dview.push(dict(numer_L=numer_L))
p_result = dview.map_sync(lambda y: numpy.linalg.det(numer_L(3,y)), range(30))
p_result

it may be that IPython is being too smart for you. What if you also do from numpy import matrix as ImmutableMatrix in your original namespace?

Sorry if this is a non-answer, but it wouldn't fit into a comment.

Answer 2

I don't know if this is the best answer, but it is my current workaround. I inspected the lambdify source code and found a lambdastr function that ought to show me the code being generated:

from sympy.utilities.lambdify import lambdastr
lstr = lambdastr((x,y), L, dummify=True)

Here is what the generated code, lstr, looks like:

'lambda x,y: (ImmutableMatrix([[y, 1, 2, y, 4, 5, y, 7, 8], [0, x, y, 3*x, 4*x, y, 6*x, 7*x, y], [0, y, 2*x**2, 3*x**2, y, 5*x**2, 6*x**2, y, 8*x**2], [y, x**3, 2*x**3, y, 4*x**3, 5*x**3, y, 7*x**3, 8*x**3], [0, x**4, y, 3*x**4, 4*x**4, y, 6*x**4, 7*x**4, y], [0, y, 2*x**5, 3*x**5, y, 5*x**5, 6*x**5, y, 8*x**5], [y, x**6, 2*x**6, y, 4*x**6, 5*x**6, y, 7*x**6, 8*x**6], [0, x**7, y, 3*x**7, 4*x**7, y, 6*x**7, 7*x**7, y], [0, y, 2*x**8, 3*x**8, y, 5*x**8, 6*x**8, y, 8*x**8]]))'

It looks like this ought to work if I just import numpy.matrix as ImmutableMatrix, but no dice:

# in parallel
# do imports and push our lambda function over
dview.execute('import numpy')
dview.execute('from numpy import matrix as ImmutableMatrix')
dview.push(dict(numer_L=numer_L))
p_result = dview.map_sync(lambda y: numpy.linalg.det(numer_L(3,y)), range(30))

Fails with the same error as before.

I got it to work by pushing the generated code as a string, and doing the eval myself:

# in parallel
# do imports and push our lambda function over
dview.execute('import numpy')
dview.execute('from numpy import matrix as ImmutableMatrix')
dview.push(dict(lstr=lstr))
p_result = dview.map_sync(lambda y: numpy.linalg.det(eval(lstr)(3,y)), range(30))

This works, and matches the result computed sequentially

p_result == s_result  # True

So, I kind of got things working, but it doesn't feel like the right way to do it, probably due to the way lambdify works.