CODEWITHSUNDEEP
pythonmultiprocessingpython-multiprocessing
user2909415
user2909415

Reputation: 1029

Python: How to share an Unpickable Object

I am doing some multiprocessing and I need to share an instance between two processes. I have been trying to use the multiprocessing module to try and accomplish this but there isn't much hope to share anything that can't be pickled it seems; so far I have tried to use a manager and create a proxy object to handle my object by following this SO question and this SO question. I understand that sharing mutable object instances isn't exactly python's forte, but what is the easiest way to do this?

To layout my situation more clearly, I am working on UNIX systems exclusively so it uses forks and copy-on-write memory management to my understanding. This object that I need to share is read-only on the main process but read and write on the subprocess. The easiest way I can think of doing this is to just share a reference to the object instance in memory to the subprocess and communicate between processes not to write when the object is being used by the main process.

Upvotes: 4

Views: 2048

Answers (1)

user2909415
user2909415

Reputation: 1029

I solved this problem myself, but for anyone who comes to this thread with the same problem:

I made the mistake of instantiating a SyncManager() object then calling the BaseManager.register() method assuming this would work since SyncManager is just a subclass of BaseManager(). I didn't look into exactly why this is the case, but I do know the fix, and that is to just instantiate a BaseManager() and register your Python object with that instead.

Upvotes: 1

Related Questions