CODEWITHSUNDEEP
javakubernetesapache-kafkaspring-kafkakafka-producer-api
Denis
Denis

Reputation: 3707

How to choose Kafka transaction id for several applications, hosted in Kubernetes?

I have a classic microservice architecture. So, there are differ applications. Each application may have 1..N instances. The system is deployed to Kubernetes. So, we have many differ PODs, which can start and stop in any time.

I want to implement read-process-write pattern, so I need Kafka transactions.

To configure transactions, I need to set some transaction id for each Kafka producer. (Actually, I need transaction-id-prefix, because of I use Spring for my applications, and it has such API). These IDs have to be the same, after application is restarted.

So, how to choose Kafka transaction id for several applications, hosted in Kubernetes?

Upvotes: 4

Views: 6585

Answers (1)

Gary Russell
Gary Russell

Reputation: 174494

If the consumer starts the transaction (read-process-write) then the transaction id prefix must be the same for all instances of the same app (so that zombie fencing works correctly after a rebalance). The actual transaction id used is <prefix><group>.<topic>.<partition>.

If you have multiple apps, they should have unique prefixes (although if they consume from different topics, they will be unique anyway).

For producer-only transactions, the prefix must be unique in each instance (to prevent kafka fencing the producers).

EDIT

Note that KIP-447 changed all this; it is no longer necessary (when using EOSMode.V2 aka BETA) to keep the tranactional id the same - consumer metadata is used for fencing instead.

Upvotes: 6

Related Questions