CODEWITHSUNDEEP
rabbitmqmessage-queuepython-multiprocessing
simon
simon

Reputation: 2831

why use rabbitmq or similar versus python builtin multiprocessing queue?

I have a producer of tasks and multiple workers to consume those tasks. Many places recommend rabbitmq and/or celery. However python has a builtin multiprocessing queue that can be shared on an ip/port using a manager/proxy. What would be the advantages of using something like rabbitmq instead?

Upvotes: 3

Views: 1537

Answers (1)

StuartLC
StuartLC

Reputation: 107347

RabbitMq is an enterprise level tool, typically deployed separately on out-of-process servers / VMs / Containers, and plays in the enterprise service bus space.

  • Rabbit has reliable messaging as an objective - e.g. messages are persisted, and nodes in the cluster can be restarted without losing messages.
  • Supports a large range of messaging topologies, such as Point-Point, Fan out, and Topic subscriptions
  • Can be scaled for volume by adding multiple nodes to a cluster
  • Allows for conditional routing of messages to queues using routing keys or header filters
  • Agnostic of client technology, i.e. Clients can be on any platform which support the AMQP protocol
  • Has an out of the box administration, monitoring and diagnostics UI
  • Has a wide range of extensions and tools, such as shovels allowing messages to be replicated across multiple RabbitMQ clusters.

I'm no Python expert, but from what I understand of the multiprocessing package, it serves as an manager for distributing work between worker processes and threads, so IMO would be regarded as a more local system concern, as opposed to 'enterprise' level. e.g. you would need to handle persistence, i.e. so messages are not lost during a crash / restart, and would likely need to built your own administration and monitoring tools.

Upvotes: 4

Related Questions