Statically link python37.dll and vcruntime140.dll when using cython --embed

Question

Let's say I'm "cythonizing" this test.py:

import json
print(json.dumps({'key': 'hello world'}))

with:

cython test.py --embed
call "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x64
cl test.c /I C:\Python37\include /link C:\Python37\libs\python37.lib

As mentioned in Minimal set of files required to distribute an embed-Cython-compiled code and make it work on any machine, it is necessary to distribute python37.dll and vcruntime140.dll and the content of Lib\ (either as Lib\ or packed into a python37.zip) as well, along the test.exe file.

Question: How to modify the cl.exe ... command to ask the compiler to statically link python37.dll and vcruntime140.dll inside the test.exe file?

(so that shipping python37.dll and vcruntime140.dll separately is no longer necessary)

Answer 1

Remark: There is probably a better/saner/easier option than the one presented in bellow.

The main difference between these two approaches: while in this approach all C-extension must be backed in into the resulting executable, in the alternative approach C-extension are compiled separately or an additional C-extension could also be added later on to the distribution.

---

While creating a statically linked embeded-Python-executable is relatively easy on Linux (see for example this SO-post) it is much more complicated on Windows. And you probably don't want to do it.

Also the result might not be what one would expect: Due to limitations of dlls compared to Linux' shared objects, the resulting static python-version will not be able to use/load any other c-extensions, as the one backed-in during the compile/link time (Note: this is not entirely true, a workaround is presented in this answer).

I also would not recommend to switch from vcruntime-dll to its static version - it would only make sense, when everything (exe, c-extensions, other dll which depend on vcruntime) is statically linked into one huge executable.

The first stumbling block: While on Linux python distributions often have a static Python-library already shipped, Windows distributions have only the dll, which cannot be statically linked in.

Thus one needs to build a static library on Windows. A good starting point is this link.

After downloading the source for the right Python version (git clone --depth=1 --branch v3.8.0 https://github.com/python/cpython.git) you can go to cpython\PCBuild and build cpython as explained in documentation (which might vary from version to version).

In my case it was

cd cpython/PCbuild
.\build.bat -e -p x64 

No we have a functioning Python3.8 installation, which can be found in cpython/PCbuild/amd64. Create folder cpython/PCbuild/static_amd64 and add the following pyx-file:

#hello.pyx
print("I'm standalone")

copy python38.dll to static_amd64 for the time being.

Now let's build our program with embedded python interpreter:

cython --embed -3 hello.pyx
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x64
cl /c hello.c /Fohello.obj  /nologo /Ox /W3 /GL /DNDEBUG /MD -I<path_to_code>\cpython\include -I<path_to_code>\cpython\PC
link hello.obj python38.lib  /OUT:hello_prog.exe /nologo "/LIBPATH:<path_to_code>\cpython\PCbuild\amd64"

After the start, hello_prog.exe lies to us, as it is not really standalone. The good news is: it finds the Python-installation which is needed as described for example here.

Now let's create a static python38-library. For that we open pcbuild.sln in cpython/PCbuild-folder and change pythoncore-project's setting to produce static library in PCbuild\amd64_static-folder. Rebuild it.

Now we can build the embedded-python-exe:

cl /c hello.c /Fohello.obj /D "Py_NO_ENABLE_SHARED" /nologo /Ox /W3 /GL /DNDEBUG /MD -I<path_to_code>\cpython\include -I<path_to_code>\cpython\PC
link hello.obj python38.lib "version.lib" "shlwapi.lib" "ws2_32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /OUT:hello_prog.exe /nologo "/LIBPATH:<path_to_code>\cpython\PCbuild\static_amd64"

Compared to the build against dll we had to change the following:

• Py_NO_ENABLE_SHARED (i.e. /D "Py_NO_ENABLE_SHARED") is added to preprocessor definitions, otherwise the linker will look for wrong symbols.
• the Windows dependencies (i.e. version.lib and so on) which were brought by the python-dll need to be passed to the linker explicitly now (this can be looked up in the linker-command-line of the pythoncore-project).
• lib path show to the static folder, i.e. "/LIBPATH:<path_to_code>\cpython\PCbuild\static_amd64" now.
• there might be other smaller troubles (different optimization levels, link-time-code generation, disabling whole-program-optimization and so on) depending on your exact tool chain.

We can delete the python38.dll from static_amd64 now and the hello_prog.exe still works.

On Linux, this would be "mission accomplished", on Windows we are just at the beginning...

Make sure that cpython-folder has a DLLs-folder with all right pyd-files, otherwise create and copy all pyd-files from PCbuild/amd64-folder.

Let's make our pyx-file a little bit more complicated:

import _decimal
print("I'm standalone")

_decimal is a fast implementation of the decimal-module which is a C-extension and can be found in the DLL-folder.

After cythonizing and building it, running hello_prog.exe leads to the following error message:

import _decimal
ImportError: DLL load failed while importing _decimal: The specified module could not be found.

The problem is easy to find:

dumpbin /DEPENDENTS ../amd64/_decimal.pyd
...
python38.dll
... 

The extensions of our installation still depends on the python-dll. Let's rebuild them against the static library - we need to change library path to from amd64 to static_amd64, to add preprocessor define Py_NO_ENABLE_SHARED and all missing windows-libraries (i.e ""version.lib"& Co.) and adding /EXPORT:PyInit__decimal to link options, otherwise, due to Py_NO_ENABLE_SHARED it becomes invisible. The result has no dependency on the python-dll! We copy it to the DLLs-folder and ...

hello_prog.exe
# crash/stopped worked

What is happening? We violated one definition rule (ODR) and ended up with two Python-interpreters: the one from the hello_prog.exe, which is initialized and the one from _decimal.pyd which is not initialized. _decimal.pyd "speaks" to its interpreter which is not initialized and bad things happens.

The difference to Linux is the difference between shared-objects and dlls: while shared-objects can use symbols from the exe (if the exe is built with right options) dll cannot and thus must either depend on a dll (which we don't want) or need to have its own version.

To avoid the violation of ODR we have only one way out: it must be linked directly into our hello_word-executable. So let's change the project for _decimal to static library and rebuild it in static_amd64-folder. Deleting the pyd from "DLLs"-folder and adding /WHOLEARCHIVE:_decimal.lib to the linker-command-line (whole archive while otherwise the linker would just discard _decimal.lib as none of its symbols is referenced somewhere), leads to an executable, which has the following error:

ModuleNotFoundError: No module named '_decimal'

This is expected - we need to tell to the interpreter, that the module _decimal is backed in and should not be searched on the python path.

The usual solution for this problem is to use PyImport_AppendInittab just before Py_Initialize, that means we need to change the c-file generated by cython (there might be workarounds, but due to multi-phase initialization it is not that easy. So probably a saner way to embed Python is the one presented here or here were main isn't written by Cython). The c-file should look as follows:

//decalare init-functions
extern  PyObject* PyInit__decimal();
...
int main(int argc, char** argv) {
...
    if (argc && argv)
        Py_SetProgramName(argv[0]);
    PyImport_AppendInittab("_decimal", PyInit__decimal); //  HERE WE GO
                                                         //  BEFORE Py_Initialize
    
    Py_Initialize();

Building everything now leads to an exe which prints

I'm standalone

and this time it is not lying to us!

Now we have to repeat the last steps for all other built-in extension we need.

---

The above means there are some restrictions for statically built python-interpreter: All built-in modules need to be backed into the executable and we cannot extend the interpreter latter on with libraries like numpy/scipy (but can to do it directly at the compile/link time).

---

Getting rid of vcruntime-dll is easier: all above steps must be done with /MT option instead of MD-option. However, there might be some problems due to usage of other dlls (e.g. _ctypes needs ffi-dll) which where built with the dll-version (and thus we once again have ODR-violated) - so I would not recommend it.

Answer 2

This is slightly different approach (as suggested by @ssbssa's comments): the main difference is that with this version, one can add further C-extensions later on and doesn't have to back them in into the resulting executable.

The first steps until the creation of hello_prog.exe linked against static python-library are the same as in the above answer. The command

...
link hello.obj ... /OUT:hello_prog.exe ...

creates not only the the exe itself, but also the lib-file hello_prob.lib. This lib-file can be used for linkage, because exe itself exports a lot of symbols from Python-library. The behavior is similar to -Xlinker -export-dynamicbehavior when linking embedded python executable on Linux.

Now, when C-extensions are built (e.g. _decimal we need to add hello_prog.lib as linking dependency (i.e. in Properties/Linker/Input->Additional Dependencies).

When we look at the run time dependencies of _decimal.pyd, we see:

dumpbin /DEPENDENTS _decimal.pyd
...
Dump of file _decimal.pyd

File Type: DLL

  Image has the following dependencies:

    hello_prog.exe
    ...

When the resulting pyds are linked in such a way and can be found by the embedded interperter, we run hello_prog and see:

I'm standalone

which means everything works as expected.

Should further C-extension be built, hello_prob.lib-file must be provided/stored somewhere.