Compiling Python code into an exe file calling just the required DLLs

Question

There seems to be little information out there on this subject - I have tried py2exe to no avail (errors out w/ Python 3.5), as well as pyinstaller. The latter seems to include EVERY extension module installed on the system unless they are specifically excluded during build time, which ends up being 50+ on even a small build. I have also seen pyinst and cxFreeze where you can specify which packages to include and it bundles Python with it, BUT I want simple and small. All the work I do involves AT MOST NumPy, SciPy, and Pandas. Typically these can all be reduced to NumPy only functions, and most complex ones I rewrite in Cython (with memoryviews). Looking around the internet I've found that python.exe is a C++ program that only calls Python.h and loads python35.dll - this suggests something as simple as this C++ code sample would allow one to access all of Python and NumPy functionality as so:

#include "Python.h"
#include <\numpy\core\include\numpy\ required includes>

int
wmain(int argc, wchar_t **argv)
{
    wchar_t *myargs[3] = { argv[0], L"-m", L"myscript.py" };
    return Py_Main(3, myargs);
}

Example is from: https://blogs.msdn.microsoft.com/pythonengineering/2016/04/26/cpython-embeddable-zip-file/

Does anyone have a good way or a good reference to generate a small build which includes only what you need for the exe to run? Or can confirm the above would actually work with modifications? Much appreciated, I think this skill is a niche subject...

Answer 1

If you can take the dependency, you can use boost::python to help.

There's a full working exemplar app below, which only links boost_python and python2.7 shared libraries. An -O3 build is only 51K.

Import a python module:

You use boost::python::exec to execute an import statement, importing your python module into a boost::python::object

bp::object import(const std::string& module, const std::string& path)
{
    bp::object globals = bp::import("__main__").attr("__dict__");

    bp::dict locals;
    locals["module_name"] = module;
    locals["path"]        = path;

    // execute some python code which imports the file
    bp::exec("import imp\n"
             "new_module = imp.load_module(module_name, open(path), path, ('bp', 'U', imp.PY_SOURCE))\n",
             globals,
             locals);
    return locals["new_module"];
}

// get an object containing the contents of the file "test.py"
bp::object module = import("test", "test.py");

Get a handle to an element in the python module:

// get a handle to something declared inside "test.py"
bp::object Script = module.attr("Script");

Instantiate an object:

// Instantiate a script object
bp::object script = Script();

Call a member function of Script:

// calls Script.run(), passing no arguments
script.attr("run")();

You can also expose C++ code to python:

Expose a C++ class to the module you just imported using boost::python::class:

struct Foo
{
    void func();
}

bp::object FooWrapper(
    bp::class_<Foo>("Foo")
        .def("func", &Foo::func)
    );

bp::object foo    = FooWrapper(); // instantiate a python wrapped Foo object
bp::object script = Script(foo);  // create a Script instance, passing foo 

Working example:

test.py

class Script(object):
    def __init__(self, ifc):
        print 'created script'
        self.ifc = ifc

    def run(self):
        print 'running'
        self.ifc.execute(5)

    def result(self, i):
        print 'result={}'.format(i)

main.cpp

#include <boost/python.hpp>

namespace bp = boost::python;

bp::object import(const std::string& module, const std::string& path)
{
    bp::object globals = bp::import("__main__").attr("__dict__");

    bp::dict locals;
    locals["module_name"] = module;
    locals["path"]        = path;

    bp::exec("import imp\n"
             "new_module = imp.load_module(module_name, open(path), path, ('bp', 'U', imp.PY_SOURCE))\n",
             globals,
             locals);
    return locals["new_module"];
}

///////////////////////////

class Runner
{
public:
    void init(bp::object script)
    {
        // capture methods at creation time so we don't have to look them up every time we call them
        _run    = script.attr("run");
        _result = script.attr("result");
    }

    void run()
    {
        _run(); // call the script's run method
    }

    void execute(int i) // this function is called by the python script
    {
        _result(i * 2); // call the script's result method
    }

    bp::object _run;
    bp::object _result;
};

int main()
{
    Py_Initialize();

    // load our python script and extract the Script class
    bp::object module = import("test", "test.py");
    bp::object Script = module.attr("Script");

    // wrap Runner and expose some functions to python
    bp::object RunnerWrapper(
        bp::class_<Runner>("Runner")
            .def("execute", &Runner::execute)
        );

    // create a python wrapped instance of Runner, which we will pass to the script so it can call back through it
    bp::object wrapper = RunnerWrapper();
    bp::object script  = Script(wrapper);

    // extract the runner instance from the python wrapped instance
    Runner& runner = bp::extract<Runner&>(wrapper);

    // initialise with the script, so we can get handles to the script's methods we require
    runner.init(script);

    runner.run();

    Py_Finalize();
    return 0;

CMakeLists.txt

cmake_minimum_required (VERSION 3.2.2)

find_package(Boost COMPONENTS python REQUIRED)
find_package(PythonLibs 2.7 REQUIRED)

add_executable            (py_embed main.cpp)
target_link_libraries     (py_embed ${Boost_PYTHON_LIBRARY} ${PYTHON_LIBRARIES})
target_include_directories(py_embed SYSTEM PRIVATE ${PYTHON_INCLUDE_DIRS})

Downloadable source code here