CODEWITHSUNDEEP
pythonmultiprocessing
Wessie
Wessie

Reputation: 3510

Alternative to multiprocessing.manager in Python

I've been managing a program that uses multiprocessing.manager due to some requirements, however we have been getting a steady amount of errors such as timeouts, invalid references and other similar errors.

Now I'm curious if there is a more developed alternative to multiprocessing.manager that has better overall reliability and less state tracking on the client side.

I've tried Google on the subject but due to the odd combination of keywords I only receive bogus results.

Our usual use case is similar to this:

def connect():
  manager = CustomManager(address=manager_address, authkey=manager_authkey)
  manager.connect()
  session = manager.session()
  return session

connect().some_function()

Upvotes: 2

Views: 1738

Answers (1)

Marwan Alsabbagh
Marwan Alsabbagh

Reputation: 26778

Judging by the question and your comments, If you want something more solid to manage processes there are better alternatives to using the multiprocessing module. Below are two options you might want to explore:

Gearman

This is a description of the Gearman project.

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work

Instagram has workers written in python and uses Gearman to run these jobs in the background. You can read about it in the Task Queue section of this What Powers Instagram post.

Celery: Distributed Task Queue

Celery is an asynchronous task queue based on distributed message passing, it is focused on real-time operation. It is really popular in the Django community.

Both solutions are very scalable and used extensively so you will find a lot of documentation and tutorials on how to use them. They are more involved though so there will be more of an initial learning curve. But I think it might be worth the time investment if you are hitting the limit of multiprocessing.

Upvotes: 2

Related Questions