CODEWITHSUNDEEP
linuxdockerapache-kafkaapache-zookeeper
jessica
jessica

Reputation: 2610

Kafka using Docker for production clusters

We need to build a Kafka production cluster with 3-5 nodes in cluster ,

We have the following options:

  1. Kafka in Docker containers (Kafka cluster include zookeeper and schema registry on each node)

  2. Kafka cluster not using docker (Kafka cluster include zookeeper and schema registry on each node)

Since we are talking on production cluster we need good performance as we have high read/write to disks (disk size is 10T), good IO performance, etc.

So does Kafka using Docker meet the requirements for productions clusters?

more info - https://www.infoq.com/articles/apache-kafka-best-practices-to-optimize-your-deployment/

Upvotes: 3

Views: 3738

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191963

It can be done, sure. I have no personal experience with it, but if you don't otherwise have experience managing other stateful containers, I'd suggest avoiding it.

As far as "getting started" with Kafka in containers, Kubernetes is the most documented way, and Strimzi (free, optional commercial support by Lightbend) or Confluent Operator (commercial support by Confluent) can make this easy when using Kubernetes or Openshift. Or DC/OS offers a Kafka service over Mesos/Marathon. If you don't already have any of these services, then I think it's apparent that you should favor not using containers.

Bare metal or virtualized deployments would be much easier to maintain than hand-deployed containerized ones, from what I have experienced. Particularly for logging, metric gathering, and statically assigned Kafka listener mappings over the network. Confluent provides Ansible scripts for doing deployments to such environments

That isn't to say there's companies that have been successful at it, or at least tried. IBM, RedHat, and Shopify immediately pop up in my searches, for example

Here's a few talk about things to consider when Kafka is in containers https://www.confluent.io/kafka-summit-london18/kafka-in-containers-in-docker-in-kubernetes-in-the-cloud

https://kafka-summit.org/sessions/running-kafka-kubernetes-practical-guide/

Upvotes: 2

Related Questions