CODEWITHSUNDEEP
pythonmultithreadingmultiprocessingctypes
catubc
catubc

Reputation: 508

Python calling multiprocessing from separate thread

I'm trying to call a multiprocessing function from a python thread to avoid the Global Interpreter Lock affecting my multiprocessing function.

The logic looks like this:

python --- ongoing python processing...
       \
        -> py thread -> py multiprocessing call ->  C-code (dll/ctypes)

Does this make sense? Will the C code run on a separate core, or is this too complex to work?

Thanks.

EDIT: Thanks for the reply. I should clarify, but I need to call on a second thread because I have to first make a python array and then pass the pointer to a C function. So I can't call the multiprocessing function too early (and also the main python processing needs to start and continue seamlessly).

EDIT: Here's the code logic and why I can't call a 2nd process inline with main code:

main():
...
p = Process(target=save_c, args=(...,))
p.start()
p.join()   #Main thread will lock here and wait until return;

#Other code that needs to be processed follows the multiprocess call

save_c():
''' Function which calls C-module '''
_sum = ctypes.CDLL('/home/pi/murphylab_picam/temp/libsum.so')
_sum.c_function.argtypes = (ctypes.c_int, ctypes.POINTER(ctypes.c_int))
_sum.c_function(ctypes.c_int(num_numbers), array_type(*_sum.numbers))

What am I missing here? Is there a different way to use multiprocessing inline with ongoing processing?

Upvotes: 0

Views: 272

Answers (1)

noxdafox
noxdafox

Reputation: 15040

You don't need to join immediately after you create a process as long as you don't want to wait for that process to finish before continuing.

This is the principle of concurrent programming.

Important thing is that you eventually call join or your main process will terminate leaving the child orphan.

child_process = Process(....)
child_process.start()  # the child process will start its independend execution here

do_some_stuff()
do_some_more_stuff()

child_process.join()  # here you are waiting for it to finish

Upvotes: 1

Related Questions