Apache kafka message communication between microservices

Question

I have a problem, that I want to solve using kafka queues. I need to process some result, then return it to the user.

As you can see in the picture, the Rest Service, requests something to the Calculator Service.

Both services have a kafka consumer, and a kafka producer.

• The rest service receive a request, then produces a message on toAdd queue, then keep consuming the fromAdd queue, until receives a value.

• The calculator service keep consuming the toAdd queue, when some message comes, it sum two values, then produces a message on fromAdd queue.

Sometimes the rest service receives old messages from the queue, or more than one message. I find something about idempotent configuration, but I don't know how to implement right.

Is that diagram, the right way to the communication between two or more services using kafka?

Can someone give a example?

Thanks.

Answer 1

Is that diagram, the right way to the communication between two or more services using kafka?

If you mean "Does it make sense to have two or more services communicate indirectly through Kafka?", then yes, it does.

Can someone give a example?

Here are some good pointers including examples:

• Build Services on a Backbone of Events, Confluent blog, May 2017
• Commander: Better Distributed Applications through CQRS, Event Sourcing, and Immutable Logs, by Bobby Calderwood, StrangeLoop, Sep 2016
  • Recorded talk
  • Reference implementation on GitHub

Answer 2

To answer your question: There is no problem with such communication.

Now referring back to other parts... Keep in mind that it's an asynchronous communication so you should not keep HTTP connection open and keep user of that service waiting for the response. This is just not the way to go. You can solve this in many ways. For instance: you can use WebSockets, you can send an email/SMS/slack msg to the user with the reply and so on.