Bart M
Bart M

Reputation: 727

Cythonize ends with 'fatal error C1002: compiler is out of heap space in pass 2'

I try to cythonize a .py script. It is a PyQt5 gui with a large number of QToolButtons, and a working EventFilter. The c module is built successfully, however, the compilation fails with the following error:

d:\stuff\mapform2a.c(11338) : fatal error C1002: compiler is out of heap space in pass 2 LINK : fatal error LNK1257: code generation failed error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe' failed with exit status 1257

The compiler comes from Visual Studio 2019. Python 3.5.5 (yes, old, I know, but I have reasons...).

Is there any way to increase the heap space when trying to "cythonize -i script.py" ?

Cython documentation is really not clear on this (for a non-C-expert at least...)

EDIT The full log is as follows:

C:\temp\MapForm>python setup.py build_ext --inplace Compiling MapForm2A.py because it changed. [1/1] Cythonizing MapForm2A.py C:\Anaconda3\lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\temp\MapForm\MapForm2A.py tree = Parsing.p_module(s, pxd, full_module_name) running build_ext building 'MapForm2A' extension creating build creating build\temp.win-amd64-3.5 creating build\temp.win-amd64-3.5\Release C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Anaconda3\include -IC:\Anaconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" /TcMapForm2A.c /Fobuild\temp.win-amd64-3.5\Release\MapForm2A.obj MapForm2A.c creating C:\temp\MapForm\build\lib.win-amd64-3.5 C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Anaconda3\libs /LIBPATH:C:\Anaconda3\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_MapForm2A build\temp.win-amd64-3.5\Release\MapForm2A.obj /OUT:build\lib.win-amd64-3.5\MapForm2A.cp35-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.5\Release\MapForm2A.cp35-win_amd64.lib MapForm2A.obj : warning LNK4197: export 'PyInit_MapForm2A' specified multiple times; using first specification Creating library build\temp.win-amd64-3.5\Release\MapForm2A.cp35-win_amd64.lib and object build\temp.win-amd64-3.5\Release\MapForm2A.cp35-win_amd64.exp Generating code c:\temp\mapform\mapform2a.c(7545) : fatal error C1002: compiler is out of heap space in pass 2 LINK : fatal error LNK1257: code generation failed error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe' failed with exit status 1257

I can only add that the process gets stuck on the "Generating code" message for about 90 seconds before raising the C1002 exception.

The setup file is rather standard:

from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("MapForm2A.py")
)

The module is pure PyQt5 (pyuic5 output) with no other dependencies and works fine if interpreted directly with no Cython.

EDIT: SOLUTION (Maybe someone will need it). Thanks to @DavidW (discussion in the comments below).

Setup.py has to be modified in the following way:

from distutils import _msvccompiler
_msvccompiler.PLAT_TO_VCVARS['win-amd64'] = 'amd64'

from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("MapForm2A.py"),
)

The first two lines force 64-bit toolchain.

Upvotes: 2

Views: 1887

Answers (1)

DavidW
DavidW

Reputation: 30891

For the purposes of giving a little more explanation to something solved in the comments: the basic problem looks to be that you're compiling something large and complicated and MSVC has run out of memory at the linking step.

Microsoft has a page about this error which suggests a number of options, with the main one being to use a 64-bit compiler. (Note that this is independent of whether you're compiling a 32-bit or 64-bit module - it's merely the choice of compiler executable)

When compiling Python extension modules (especially with setup.py) the compiler setup is typically selected by distutils. Unfortunately, it looks like distutils chooses to force a 32-bit compiler (see https://github.com/python/cpython/blob/e488e300f5c01289c10906c2e53a8e43d6de32d8/Lib/distutils/_msvccompiler.py#L160).

My suggestion was to dig into the distutils internals at the top of setup.py (before any real setup takes place) to override this setting

from distutils import _msvccompiler
_msvccompiler.PLAT_TO_VCVARS['win-amd64'] = 'amd64'

Essentially all you're really doing is passing the option amd64 to the vcvarsall.bat script that microsoft supply to set up their compiler, thus getting a 64-bit compiler to build a 64-bit extension.

Upvotes: 4

Related Questions