%%cython_pyximport ImportError: Building module function failed: ['DistutilsPlatformError: Unable to find vcvarsall.bat\n']

Question

In the iPython Notebook I am trying to use the notebook function %%cython_pyximport to write a cython function that I can call later on in my notebook.

I want to use this command as opposed to %%cython because there seems to be quite a bit of overhead with it. For example when I profile my code I get this:

168495 function calls in 4.606 seconds

      Ordered by: internal time


   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    3.234    3.234    4.605    4.605 {_cython_magic_0ef63e1ad591c89b73223c7a86d78802.knn_alg}
    11397    0.326    0.000    0.326    0.000 {method 'reduce' of 'numpy.ufunc' objects}
      987    0.152    0.000    0.266    0.000 decomp.py:92(eig)
      987    0.118    0.000    0.138    0.000 function_base.py:3112(delete)

I'm hoping that using %%cython_pyximport will cut down time spent calling this function. If there is a better way please let me know.

So getting to my actual question - When I use %%cython_pyximport I get this error:

ImportError: Building module function failed: ['DistutilsPlatformError: Unable to find vcvarsall.bat\n']

Maybe it's related to something not being on my PATH but I'm not sure. What do I have to do to fix this?

I'm using Windows 7, Python 2.7.6 (Installed with Anaconda), Cython 0.20.1, iPython Notebook 2.1.0

EDIT: So after following @IanH 's suggestion I now have this error:

fatal error: numpy/arrayobject.h: No such file or directory

It seems like additional header files need to be included for numpy to work with pyximport. On this page https://github.com/cython/cython/wiki/InstallingOnWindows there is a mention of this error and how to solve it but am lost at how to apply this so that the %%cython_pyximport command will work in my notebook.

Answer 1

Since I have VS2015 installed, had to add new environment variable

SET VS90COMNTOOLS=%VS140COMNTOOLS%

Source https://stackoverflow.com/a/10558328/625189

Answer 2

I had this same exact problem, so my solution was instead to use:

[build] compiler=mingw32

[build_ext] compiler = mingw32

In the answer by IanH, I just chose to call gcc and cython directly. I figured I'd literally just circumnavigate around all the possible errors

1. Firstly, you need to compile cython using:

   cython_commands = ['cython', '-a', '-l', '-p', '-o', c_file_name, file_path]
   cython_feedback = subprocess.call(cython_commands)
   

2. Then you need to take that .c file and compile it by telling the compiler where to look for the python libraries.

   gcc_commands = ['gcc', '-shared', '-Wall', '-O3', '-I', py_include_dir, '-L', py_libs_dir, '-o', output_name,
                  c_file_name, '-l', a_lib]
   gcc_error = subprocess.call(gcc_commands)
   

py_include_dir: The path to the directory in python installation labeled 'include'

py_libs_dir: The path to the directory in python installation labeled 'libs'

c_file_name: The path to which you wish to ave the middleman c file

a_lib: The name of your python installation (ex. 'python34' or 'python35' or 'python27')

Answer 3

There are two different issues here. I'll first address the one you seem to care about.

Using pyximport instead of the cython magic function should not increase speed at all. Given your profiling results, it appears that the real problem here is that you are calling a NumPy function on the inside of a loop. In Cython you have to keep track of which function calls are done in C, and which are done in Python. Numpy universal functions are Python functions and they require the cost of calling a Python function.

How you would want to fix this depends entirely on what you are doing. If you can cleverly vectorize away the loop using NumPy operations, that is probably the best way, but not all problems can easily be solved that way. There are ways to call LAPACK routines from Cython, as described in this answer. If you are doing simpler operations (like summing along axes, etc), you can write a function that uses cython memoryviews to pass slices around internally in your Cython module. There is some discussion on the proper way to do that in this blog post. Doing these sorts of operations is usually a little harder in Cython, but it is still a very approachable problem.

Now, though I'm not convinced that pyximport will actually do what you want it to, I will still tell you how to get it working. The error you are seeing happens when distutils tries to use the Visual Studio compiler even when you haven't gotten everything set up for it. Anaconda, by default uses MinGW for Cython extensions, but for some reason it isn't set up to use MinGW with pyximport. That's easy to fix though. In your Python installation directory, (probably C:\Anaconda or something along those lines), there should be a file Anaconda\Lib\distutils\distutils.cfg. (Create it if it doesn't exist.) Modify it so that its contents contain both of the following options:

[build]
compiler=mingw32

[build_ext]
compiler = mingw32 

If I remember correctly, the first is already included in Anaconda. As of this writing, the second is not. You will need it there to make pyximport work.