CODEWITHSUNDEEP
pythonmultithreadingplugins
matanc1
matanc1

Reputation: 7033

Multiprocessing or Multithreading for plugin architecture in Python

I'm trying to implement a plugin architecture in Python.

I've started writing it using the Threading module where each plugin is a thread which I invoke using the Thread.start() method (since all plugins subclass BasePlugin which subclasses Thread). However I've just come across the multiprocessing module.

I'm currently wondering if I should switch to the multiprocessing module and share data using shared memory / Pipes etc...

I'd like to get other's opinions on this.

The plugin architecture I've been working on works as follows:

An event is received by the Plugin Manager. The Plugin Manager checks for all the plugins who've subscribed to that type of event. It activates them and sends them the event object (since it holds additional information). If one of the plugins is already active there is no need to spawn it (just send the event object to it).

In addition there are a few resources which belong only to one plugin at any point in time. Each plugin can request the resource (I'm not worrying about any race condition here since there won't be that many plugins active at once).

Upvotes: 0

Views: 472

Answers (1)

aychedee
aychedee

Reputation: 25609

Threads share memory with the primary process and each other. For example you can have a list that is available to all threads. An item appended to a list can be seen by other threads. But you have to be careful. You have to understand which operations on data structures are thread safe and which are not. What happens to the behaviour of your program when two threads are checking for the existence of a key in a dictionary and then writing to it?

Multiple processes do not share memory. The new process that you start gets a copy of the memory at the point where it was spawned.

Threads use less resources. But can be hard to reason about. On the other hand communication between processes is tricky. And you can't just access an arbitrary Python data structure. Which it sounds like you want to be able to do.

A badly written plugin, if it was in a thread, could crash your whole program. Whereas if it was in a separate process this wouldn't happen. Maybe that's a consideration?

Upvotes: 1

Related Questions