CODEWITHSUNDEEP
apache-kafka
CrazySynthax
CrazySynthax

Reputation: 15064

Kafka: How to retrieve a response from consumer?

I wish to describe the following scenario: I have a node.js backend application (It uses a single thread event loop). This is the general architecture of the system: Producer -> Kafka -> Consumer -> Database

Let's say that the producer sends a message to Kafka, and the purpose of this message is the make a certain query in database and retrieve the query result. However, as we all know Kafka is an asynchronous system. If the producer sends a message to Kafka, it gets a response that the message has been accepted by a Kafka broker. Kafka broker doesn't wait until the consumer polls the message and processes it.

In this case, how can the producer get the query result operated on the database?

Upvotes: 6

Views: 3603

Answers (2)

nortontgueno
nortontgueno

Reputation: 2821

The flow using Kafka will look like this:

Kafka flow

The only way of the Producer A be aware of what happened with the message consumed by the Consumer A is producing another message. Which will be handled accordingly by any other consumer available (in this case, Consumer B).

As you already mentioned, this flow is asynchronous. This can be useful when you have a very heavy processing on your query, like a report generation or something like that, and the second producer will notify an user inbox for example.

If that is not the case, perhaps you should use HTTP, which is synchronous and you will have the response at the end of processing.

Upvotes: 5

Javier Gonzalez Benito
Javier Gonzalez Benito

Reputation: 362

You must generate new flow for communicate the query result: 

Consumer (now its a producer) -> Kafka topic -> Producer (now its a consumer)

You should consider using another synchronous communication mechanism like HTTP.

Upvotes: 1

Related Questions