CODEWITHSUNDEEP
apache-kafkakafka-consumer-apikafka-producer-api
Achilleus
Achilleus

Reputation: 1944

When to close a producer or consumer

Lately, we are having some performance issues with our Kafka consumers and producers. We use Kafka Java API in scala. What is considered to be good practice wrt opening and closing of consumer and producer objects? I believe this is a quite open-ended question and the right answer is always depends but I am trying to reason about this.

Can Consumers can be long-running connections and left open?

Should producers be closed whenever we are done producing messages?

Upvotes: 6

Views: 3914

Answers (1)

Adam Kotwasinski
Adam Kotwasinski

Reputation: 4554

Can Consumers can be long-running connections and left open?

In general, yes.

In detail: depending on your consumer configuration.

If your consumers are members of consumer group they certainly should be closed - to trigger the rebalance at earliest possible time.

If your consumers are using auto-commiting of offsets, they would still keep committing every N ms (AFAIK 60k), possibly wasting resources.

Otherwise, they can stay - but why waste resources?

Should producers be closed whenever we are done producing messages?

In general, yes.

Depends on your design, but if you can say at certain time you won't be sending any more messages, then you can close. That does not mean you should be closing and re-creating a producer after every sent message.

Upvotes: 2

Related Questions