CODEWITHSUNDEEP
pythonmingwcythonimporterrordistutils
user3924729
user3924729

Reputation: 29

Importing cython file in python, using pyximport results in the ImportError: because of distutils.error.CompilerError

I have forked scikit-learn github code from https://github.com/scikit-learn/scikit-learn. I want to debug this file: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/cluster/_mean_shift.py .

I have added

from sklearn.datasets.samples_generator import make_blobs
bandwidth=0.5
num_mean_shift_iterations=5
num_points=100
centers = [[-1, -1], [-1, 1], [1, -1], [1, 1]]
x1, y = make_blobs(n_samples=num_points, centers=centers, cluster_std=0.4, random_state=42)
clustering = MeanShift(bandwidth=bandwidth, n_jobs = -1, max_iter=num_mean_shift_iterations).fit(x1)

in _mean_shift.py. And I am running it using a Sypder environment with Python 3.7 installed on it.

Following is the complete error.

ImportError: Building module utils.murmurhash failed ["distutils.errors.CompileError:command 'C:\\\\Program Files (x86)\\\\Microsoft Visual Studio\\\\2019\\\\Community\\\\VC\\\\Tools\\\\MSVC\\\\14.24.28314\\\\bin\\\\HostX86\\\\x64\\\\cl.exe' failed with exit status 2\n"]

where utils.murmurhash is what I am trying to import in python file located at different location.

I have created a new environment in python using: https://pystan.readthedocs.io/en/latest/windows.html i.e. all the libraries listed on this webpage are installed.

distutils.cfg that contains the following code is located at: C:\Users\username\Anaconda3\envs\stan_env\Lib\distutils\distutils.cfg

[build]
compiler=mingw32

[build_ext]
compiler = mingw32

I have used:

import pyximport
pyximport.install()

in python file before importing cython_files.

Please point me out to the error. Thanks.

Upvotes: 0

Views: 1085

Answers (1)

MakowToms
MakowToms

Reputation: 19

As I see you you are using:

import pyximport
pyximport.install()

to compile Cython in order to import *.pyx files.

On the site it is written: "If your module doesn’t require any extra C libraries or a special build setup, then you can use the pyximport module"

Probably it can cause a problem, because sklearn imports many C libraries.

In order to import *.pyx first you have to compile it by running:

# change directory to forked project ([example-how-to-do-that][2])
# only for Windows users with anaconda:
conda activate {environment_name} # the same name which you provided during environment creation
# only for linux users who don't use anaconda:
# source {environment_name}/bin/activate
python setup.py build_ext --inplace

in terminal.

I also had problems when I tried to compile Cython modules directly in the code (by pyximport package). I cannot import them. In my case it was also files from sklearn. And the method with compiling Cython by terminal helped me.

Upvotes: 2

Related Questions