CODEWITHSUNDEEP
node.jseventsarchitecturemicroservicesmessage-queue
BigL
BigL

Reputation: 165

Internal Message Queue vs External Message Queue

I am currently working on a microservice system using node and python. We are currently using AWS SQS to process messages between services on queues however in one of the services we send a message to the queue and that same service processes that message. no other microservice handles it. So my question is is it better to handle internal messages through an event bus/event queue inside the same application or container or should the messages be sent to an external service via a network request and processed by a worker.

  1. Internal Case Container
Application
|-------------------------------------------------------|
|                                                       |
|  |---------|         Send Message          |---------||
|  | Main    |------------------------------>| Internal||
|  |  App    |<------------------------------| Queue   ||
|  |---------|         Emit Message          |---------||
|                                                       |
|-------------------------------------------------------|
  1. External Queue
Application
|-----------------|
|                 |
|                 |     Send Network Message  |---------|
|                 |-------------------------->| External|
|                 |<--------------------------| Queue   |
|                 |   Process Network Message |---------|
|                 |                                      
|-----------------|

Upvotes: 1

Views: 1461

Answers (1)

Cosmin Ioniță
Cosmin Ioniță

Reputation: 4055

There are a couple of aspects I would think of before taking a decision:

  • Considering the internal queue, what happens if the entire pod/box crashes (for whatever reason)? It is acceptable from a business and architectural perspective to lose the messages stored in that queue?
  • Is latency an important vector in your data processing workflows? If yes, then it might be an option to use the internal queue, since you're not going off-process with the data. If not, and data consistency is more important, then an external queue might be the way to go.
  • From scalability perspective, what if the internal queue gets too big? Eventually it may fill out the memory on the box/pod, but also you may want to use other workers to keep the processing latency within some thresholds (so you still may end up going off process).
  • Also, let's say that your process that consumes from the queue crashes (for some reason), or it has a logical bug, but the queue still has data in it, it might be challenging to replay it, since the queue is on the same process with the app. With an external queue, you just plug in a new working consumer, that might be on a different box.
  • From an observability perspective (numbers of messages consumed/sent/in-flight, etc) it's way better to use an external queue, than to manage all that kind of stuff by yourself.

TL;DR - if you have latency constraints and the queue is used to keep a small number of messages that are consumed fast, then it might make sense to use an internal queue. Otherwise, (and considering the above), an external queue is a better option.

Upvotes: 3

Related Questions