CODEWITHSUNDEEP
javaapache-kafkakafka-consumer-apikafka-producer-api
Siddharth Gupta
Siddharth Gupta

Reputation: 1613

Communicating to multiple kafka clusters of different versions

I have two kafka clusters with versions 0.8.2.1 and 0.10.1.

I want to consume some events from 0.8.2.1 and produce others to 0.10.1. I have been reading about the backward compatibility, and understand that clients are forward compatible. I couldn't find any reference to 0.8.2.1 producing to a future version. Is it possible, to use a single client and communicate to both the cluster.

Is there a way I can use both the clients in my same java code. I am using maven as my build tool.

Upvotes: 2

Views: 1380

Answers (1)

Matthias J. Sax
Matthias J. Sax

Reputation: 62285

If you want to read/write to two Kafka Clusters with different versions, you can use clients that correspond to the lower version.

For example, if you read from 0.8.2 and write to 0.10.1 you can use 0.8.2 consumer and 0.8.2 producer, because 0.8.2 producer is forward compatible with 0.10.1.

It would not work to use any higher version for the clients, as your consumer with higher version than 0.8.2 would not be able to read from 0.8.2 (even if 0.9 or 0.10.1 producer is able to write to 0.10.1 cluster).

The point is, that you cannot put a 0.8.2 consumer and 0.10.1 producer into the same app easily, as there would be class loading conflicts. It's not impossible thought, but you would need to use different class loader for consumer and producer to isolate both parts of your code to avoid class loading conflict. Done write, it would allow you to use a 0.8.2 consumer to read from 0.8.2 cluster while using 0.10.1 producer to write to 0.10.1 cluster.

But if you don't need any special feature 0.10.1 producer offers, it seems to be easier to just go with 0.8.2 for consumer and producer and avoid the hassle to handle different class loader. Custom class loaders are more tricky to use correctly as it seems.

Upvotes: 2

Related Questions