CODEWITHSUNDEEP
pythoncmultithreadingmultiprocessingpython-c-api
user5538922
user5538922

Reputation:

Can't kill multiprocessing pool in a multithreaded C application embedding Python

OS: linux
Python version: 3.6

I'm trying to extend a C application with Python runtime. The C application uses pthread and I tried to use multiprocessing forkserver in Python runtime but faced a problem. When I try to kill the program with SIGINT signal (by hitting Ctrl+C in the terminal) the worker processes get killed but the main program hangs.

Here is a toy program that produces the same problem.

#include <Python.h>
#include <pthread.h>

void * thread_start(void *unsed)
{
    PyObject *fs_mod = PyImport_AddModule("fs");
    PyObject *apply_fn = PyObject_GetAttrString(fs_mod, "apply");
    PyObject *job_fn = PyObject_GetAttrString(fs_mod, "job");
    PyObject *job_args = Py_BuildValue("()");
    PyObject_CallFunctionObjArgs(apply_fn, job_fn, job_args, NULL);
    printf("finished\n");
    return NULL;
}

int main(){
    Py_Initialize();
    PyRun_SimpleString(
        "import sys; sys.path.append('...');"
        "sys.argv=['a.out'];"  // prepare a dummy argument to avoid error in forkserver
        "import fs\n"
        "if __name__ == '__main__': fs.init()");

    while(1){
        pthread_t thread;
        pthread_create(&thread, 0, &thread_start, NULL);
        printf("joing\n");
        pthread_join(thread, 0);
    }
}

import multiprocessing as mp

pool = None


def job():
    import time
    print("running..")
    time.sleep(5)

def init():
    global pool
    mp.set_start_method('forkserver')
    pool = mp.Pool(1)

def apply(*args):
    global pool
    return pool.apply(*args)

I don't exactly know how Linux signal works. I tried to catch SIGINT signal in the main python process with signal module, but it seems that main it doesn't get to receive the signal. How would I be able to make this application die gracefully on SIGINT without hanging forever?

By reading ViKiG answer, I realized that I can first catch the KeyboardInterrupt(or SIGINT) exception in the worker processes and send some sentinel value to the main process to notify the exception and shut down the application.

After skimming through the CPython forkserver implementation, I potentially concluded that the author of the library intentionally made the main process ignore the SIGINT. I guess, currently, that the recommended way is to catch the exception in the worker processes, not in the main one.

Upvotes: 2

Views: 429

Answers (3)

georgexsh
georgexsh

Reputation: 16624

Py_Initialize() will install python's own singal handler, call Py_InitializeEx(0) instead:

void Py_InitializeEx(int initsigs)

This function works like Py_Initialize() if initsigs is 1. If initsigs is 0, it skips initialization registration of signal handlers, which might be useful when Python is embedded.

see more on its doc, and cpython source.

Upvotes: 1

user5538922
user5538922

Reputation:

It turned out that I don't have to catch the exception in the main process. I solved the problem by catching the KeyboardInterrupt(or SIGINT) exception in the worker processes and send some sentinel value to the main process to notify the exception and shut down the application.

import multiprocessing as mp


pool = None


def job():
    try:
        import time
        print("running..")
        time.sleep(5)
        return True
    except KeyboardInterrupt:
        print("Exiting..")
        return False
...

def apply(*args):
    global pool
    ret = pool.apply(*args)
    if ret:
        return pool.apply(*args)
    else:
        print("Gracefully die")

Upvotes: 0

ViKiG
ViKiG

Reputation: 783

I changed job function to handle CTRL+C interrupts:

    def job():
        import time
        try:    
            while True:
                print("running..")
                time.sleep(5)
        except KeyboardInterrupt:
            print 'Exiting job..'

My test program is exiting cleanly after above change.

AFTER EDIT:

I added this to my C program

    #include<signal.h>

    void handler() {printf("Exiting main.."); exit(0);}

Modified main as:

    int main() {
        signal(SIGINT, handler);

Upvotes: 0

Related Questions