CODEWITHSUNDEEP
amazon-web-servicesapache-kafkaapache-zookeeperaws-mskkraft
pragmatic
pragmatic

Reputation: 483

Can I use Kafka in KRaft mode on AWS MSK?

I would like to use Kafka in KRaft mode instead of Zookeeper on AWS, since Kraft mode seems to be the future of Kafka. As I understand, basic solution from AWS for Kafka is AWS MSK. But in MSK documentation I see that they mention that they use Zookeeper under the hood of their infrastructure - https://aws.amazon.com/msk/faqs/ :

... With a few clicks in the console, you can create an Amazon MSK cluster. From there, Amazon MSK replaces unhealthy brokers, automatically replicates data for high availability, manages Apache ZooKeeper nodes, automatically deploys hardware patches as needed, manages the integrations with AWS services, makes important metrics visible through the console, and supports Apache Kafka version upgrades so you can take advantage of improvements to the open-source version of Apache Kafka. ...

So is there a way to use AWS MSK with Kafka in KRaft mode?

I suppose, that I can run Kafka in KRaft mode during the Create Topic step in AWS MSK guide, but I am not sure that the whole infrastructure will be set up and work the proper way after that.

I am new with AWS MSK and will be appreciate for your help. Probably I need to use some other solutions from AWS for this task. Global idea is to use Kafka in my Java based application with microservices deployed on AWS.

Upvotes: 1

Views: 1466

Answers (3)

Andreas Berheim Brudin
Andreas Berheim Brudin

Reputation: 2290

For reference, it is actually possible for new clusters from version 3.7.x.

https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.html#3.7.kraft

It is not possible to update a cluster using Zookeeper to KRaft yet, though.

Upvotes: 1

EdbE
EdbE

Reputation: 391

KRaft is not supported in Amazon MSK as of now.

Modern versions of Kafka brokers and clients allow you to use bootstrap brokers instead of zookeeper URL to connect to Kafka cluster. This is transparent to whether ZK used or not.

As a user of managed service, you should not be affected by a choice of metadata management and election mechanism used along with the Kafka cluster, as ZK is fully managed and doesn't require any admin work by the user.

Upvotes: 0

Setanta
Setanta

Reputation: 996

No, Kraft is not supported in MSK yet. As of 3.3.2 it still uses zookeeper:

https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.html#3.3.2

Upvotes: 0

Related Questions