CODEWITHSUNDEEP
python-2.7ipythoncythonpython-importipython-parallel
baumgast
baumgast

Reputation: 11

ipyparallel imports: import own module created with cython

I have a problem in synchronising imports between ipyparallel engines. I want to run stochastic simulations in parallel. For this I have written a function named 'gillespie'. I have two versions of the function, one using Python and to gain some speed one in Cython. Both are working.

The problem now is how to make the compiled cython function available to all engines. The files 'gillespie.c', 'gillespie.so' and 'gillespie.pyx' are all in the current directory together with the notebook from where I want to use the functions. I wanted to import it like this

import gillespie as gc
%px import gillespie as gc

But this gives me:

Traceback (most recent call last)<ipython-input-50-801274ebf64a> in <module>()
----> 1 import gillespie as gc
ImportError: No module named gillespie

Just doing

import gillespie as gc

works just fine and I can use the compiled function.

Using the %px magic to import numpy or scipy to the engines works just fine.

How can I make my imports available to all engines?

Thanks for any help! baumgast

Upvotes: 1

Views: 1001

Answers (1)

minrk
minrk

Reputation: 38608

%px import gillespie executes the import statement on the engines. For that to work, you need to make sure that gillespie.so is available to those engines. You can either 'install' it by placing it in site-packages or on PYTHONPATH, or you can rely on the current-directory-based loading that you are already relying on in the notebook, in which case you need to make sure that gillespie.so is in the working directory of the engines. If you are on a single machine or shared filesystem, you can accomplish this with:

%px cd /path/to/dir/containing/gillespie.so

or send gillespie.so to the machines, if the filesystem is not shared.

Upvotes: 2

Related Questions