CODEWITHSUNDEEP
apache-kafka
Massimiliano
Massimiliano

Reputation: 665

How to configure Kafka RPC caller topic and group

I'm trying to implement an RPC architecture using Kafka as a message broker. The decision of using Kafka instead of another message broker solution is dictated by the current context.

The actual implementation consists on two different types of service:

The request/response messages published in the topics are related by the message key.

The receiver implementation was fairly simple: at startup, it creates the "request" and "response" topic, then starts consuming the request topic with the service group id (many instances of the receiver will share the same group id in order to implement a proper request balance). When a request arrives, the service processes the request and then publish the response in the response topic.

My problem is with the caller implementation, in particular while consuming the response from the response queue.

With the following assumptions:

  1. The HTTP requests must be managed concurrently;
  2. There could be more than one instance of this caller service.

every single thread/service must receive all the messages in the response topic, in order to find the message with the corresponding request key.

As an example, imagine that two receiver services produce two messages with keys 1 and 2 respectively. These messages will be published in the receiver topic, and processed. The response will then be published in the topic receiver-responses. If the two receiver services share the same group-id, it could be that response 1 arrives to the service that published message 2 and vice versa, resulting in a HTTP timeout.

To avoid this problem, I've managed to think these possible solutions:

  1. Creating a new group for every request (EDIT: but a group cannot be deleted via code, hence it would be necessary another service to clean the zookeeper from these groups);
  2. Creating a new topic for every request, then delete it afterwards.

Hoping that I made myself sufficiently clear - I must admit I am a beginner to Kafka - my question would be:

Which solution is more costly than the other? Or is there another topic/group configuration that could achieve the assumption 3?

Thanks.

Upvotes: 1

Views: 926

Answers (1)

Massimiliano
Massimiliano

Reputation: 665

I think I've found a possible solution. A group will be automatically deleted by the zookeeper when it's offset doesn't update for a period of time, determined by the configuration offsets.topic.retention.minutes.

The offset update time check should be possible to set up by setting the configuration offsets.retention.check.interval.ms.

This way, when a consumer connects to the response topic searching for the reply message, the created group can be abandoned, and it will be deleted by the zookeeper later in time.

Upvotes: 1

Related Questions