CODEWITHSUNDEEP
pythonc++embed
João Pereira
João Pereira

Reputation: 713

Stop embedded python

I'm embedding python in a C++ plug-in. The plug-in calls a python algorithm dozens of times during each session, each time sending the algorithm different data. So far so good

But now I have a problem: The algorithm takes sometimes minutes to solve and to return a solution, and during that time often the conditions change making that solution irrelevant. So, what I want is to stop the running of the algorithm at any moment, and run it immediately after with other set of data.

Here's the C++ code for embedding python that I have so far:

void py_embed (void*data){


counter_thread=false;

PyObject *pName, *pModule, *pDict, *pFunc;

//To inform the interpreter about paths to Python run-time libraries
Py_SetProgramName(arg->argv[0]);

if(!gil_init){
    gil_init=1;
    PyEval_InitThreads();
    PyEval_SaveThread();
}
PyGILState_STATE gstate = PyGILState_Ensure();

// Build the name object
pName = PyString_FromString(arg->argv[1]);
if( !pName ){
    textfile3<<"Can't build the object "<<endl;
}

// Load the module object
pModule = PyImport_Import(pName);
if( !pModule ){
    textfile3<<"Can't import the module "<<endl;
}

// pDict is a borrowed reference 
pDict = PyModule_GetDict(pModule);
if( !pDict ){
    textfile3<<"Can't get the dict"<<endl;
}

// pFunc is also a borrowed reference 
pFunc = PyDict_GetItemString(pDict, arg->argv[2]);
if( !pFunc || !PyCallable_Check(pFunc) ){
    textfile3<<"Can't get the function"<<endl;
}

/*Call the algorithm and treat the data that is returned from it
 ...
 ...
 */

// Clean up
Py_XDECREF(pArgs2);
Py_XDECREF(pValue2);
Py_DECREF(pModule);
Py_DECREF(pName);

PyGILState_Release(gstate);

counter_thread=true;
_endthread();

};

Edit: The python's algorithm is not my work and I shouldn't change it

Upvotes: 4

Views: 2510

Answers (4)

user590028
user590028

Reputation: 11730

I've been thinking about this problem, and I agree that sub interpreters may provide you one possible solution https://docs.python.org/2/c-api/init.html#sub-interpreter-support. It supports calls for creating new interpreters and ending existing ones. The bugs & caveats sections describes some issues that depending on your architecture may or may not present a problem.

Another possible solution is to use the python multiprocessing module, and within your worker thread test a global variable (something like time_to_die). Then from the parent, you grab the GIL, set the variable, release the GIL and wait for the child to finish.

But then another idea ocurred to me. Why not just use fork(), init your python interpreter in the child and when the parent decides it's time for the python thread to end, just kill it. Something like this:

void process() {

    int pid = fork();
    if (pid) {
        // in parent
        sleep(60);
        kill(pid, 9);   
        }
    else{
        // in child
        Py_Initialize();
        PyRun_SimpleString("# insert long running python calculation");
        }
    }

(This example assumes *nix, if you're on windows, substitute CreateProcess()/TerminateProcess())

Upvotes: 3

Jo&#227;o Pereira
Jo&#227;o Pereira

Reputation: 713

So, I finally thought of a solution (more of a workaround really).

Instead of terminating the thread that is running the algorithm - let's call it T1 -, I create another one -T2 - with the set of data that is relevant at that time.

In every thread i do this:

thread_counter+=1; //global variable
int thisthread=thread_counter;

and after the solution from python is given I just verify which is the most "recent", the one from T1 or from T2:

if(thisthread==thread_counter){
     /*save the solution and treat it */
}

Is terms of computer effort this is not the best solution obviously, but it serves my purposes.

Thank you for the help guys

Upvotes: 2

Yakk - Adam Nevraumont
Yakk - Adam Nevraumont

Reputation: 275385

This is based off of a cursory knowledge of python, and reading the python docs quickly.

PyThreadState_SetAsyncExc lets you inject an exception into a running python thread.

Run your python interpreter in some thread. In another thread, PyGILState_STATE then PyThreadState_SetAsyncExc into the main thread. (This may require some precursor work to teach the python interpreter about the 2nd thread).

Unless the python code you are running is full of "catch alls", this should cause it to terminate execution.

You can also look into the code to create python sub-interpreters, which would let you start up a new script while the old one shuts down.

Py_AddPendingCall is also tempting to use, but there are enough warnings around it maybe not.

Upvotes: 6

user430051
user430051

Reputation: 159

Sorry, but your choices are short. You can either change the python code (ok, plugin - not an option) or run it on another PROCESS (with some nice ipc between). Then you can use the system api to wipe it out.

Upvotes: 5

Related Questions