Building Cython module to run in C - ModuleNotFoundError

Question

I have a Cython module that I would like to call and run in C, but I am getting a ModuleNotFound error. I am following the instructions in the docs and other SO questions to the best of my knowledge, but there are limited examples of using Cython to convert Python to C beyond simple 1-file examples (i.e., non-modular).

I am running Python 3.6 on Ubuntu (via Windows 10 WSL1).

Here is a simplified example that shows the problem:

tree:

├── Makefile
├── main.c
├── main_cy.pyx
├── mod_dir
│   ├── __init__.pxd    #empty
│   ├── mod.pxd
│   └── mod.pyx
└── setup_cy.py

mod_dir/mod.pxd:

cdef public int say_hello_from_mod() except -1

mod_dir/mod.pyx:

TEXT_TO_SAY = 'Hello from Python!'

cdef public int say_hello_from_mod() except -1:
    print(TEXT_TO_SAY)
    return 0

main_cy.pyx:

print('hello from main_cy')
from mod_dir.mod cimport say_hello_from_mod

cdef public int say_hello_from_main_cy() except -1:
    say_hello_from_mod()
    return 0

main.c:

#include "main_cy.h"

int main(int argc, char *argv[])
{
    PyObject *pmodule;
    wchar_t *program;

    program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) {
        fprintf(stderr, "Fatal error: cannot decode argv[0], got %d arguments\n", argc);
        exit(1);
    }

    /* Add a built-in module, before Py_Initialize */
    if (PyImport_AppendInittab("main_cy", PyInit_main_cy) == -1) {
        fprintf(stderr, "Error: could not extend in-built modules table\n");
        exit(1);
    }

    /* Pass argv[0] to the Python interpreter */
    Py_SetProgramName(program);

    /* Initialize the Python interpreter.  Required. If this step fails, it will be a fatal error. */
    Py_Initialize();

    /* Optionally import the module; alternatively, import can be deferred until the main_cy script imports it. */
    pmodule = PyImport_ImportModule("main_cy");
    if (!pmodule) {
        PyErr_Print();
        fprintf(stderr, "Error: could not import module 'main_cy'\n");
        goto exit_with_error;
    }

    printf("Hello from C!\n");

    /* Now call into your module code. */
    if (say_hello_from_main_cy() < 0) {
        PyErr_Print();
        fprintf(stderr, "Error in Python code, exception was printed.\n");
        goto exit_with_error;
    }

    /* Clean up after using CPython. */
    PyMem_RawFree(program);
    Py_Finalize();

    return 0;

    /* Clean up in the error cases above. */
exit_with_error:
    PyMem_RawFree(program);
    Py_Finalize();
    return 1;
}

setup_cy.py:

from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize(['mod_dir/mod.pyx', 'main_cy.pyx'], annotate=False,language_level=3, include_path=['./', './mod_dir/'])
)

Makefile:

all:
    gcc -c main_cy.c -o main_cy.o <cflags>
    gcc -c main.c -o main_c.o <cflags>
    gcc main_c.o main_cy.o -o main.o <ldflags> 

When I run the Cython setup (python setup_cy.py build_ext --inplace), it successfully creates the *.c, *.h, and *.so files in the main directory and in mod_dir. Then when I run make, it runs without errors and creates all of the *.o files. However, when I run the C executable (./main.o), I get the following runtime error:

Traceback (most recent call last):
  File "main_cy.pyx", line 1, in init main_cy
    # main_cy.pyx
ModuleNotFoundError: No module named 'mod_dir'
Error: could not import module 'main_cy'

Thank you!

Answer 1

Ok I managed to find the solution here (brm's answer), but I will repost here in case others have the same question. Apparently the Python shell will automatically add the working directory to the path, but Py_Initialize() does not do this when embedding in C. To do this manually, I just had to add the line import sys\nsys.path.insert(0, '') after Py_Initialize(); in main.c. After recompiling, the executable runs as expected.