CODEWITHSUNDEEP
apache-kafka
Holm
Holm

Reputation: 3365

Kafka docker image that works without zookeeper

I read that Kafka no longer requires zookeeper, so I don't want to have zookeeper in docker-compose. But I don't know which kafka image can work w/o zookeeper. can anyone give a hint?

Upvotes: 35

Views: 29740

Answers (9)

NanoNova
NanoNova

Reputation: 953

Thanks for @daniil_'s post.

This cluster setup works for me.

version: "3"
volumes:
  broker1:
  broker2:
services:
  broker1:
    image: 'bitnami/kafka:latest'
    container_name: broker1
    environment:
      KAFKA_ENABLE_KRAFT: yes
      KAFKA_CFG_PROCESS_ROLES: 'broker,controller'
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://broker1:9092
      KAFKA_CFG_BROKER_ID: 1
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: '1@broker1:9093,2@broker2:9093'
      ALLOW_PLAINTEXT_LISTENER: yes
      KAFKA_KRAFT_CLUSTER_ID: r4zt_wrqTRuT7W2NJsB_GA
    volumes:
      - broker1:/bitnami/kafka
  broker2:
    image: 'bitnami/kafka:latest'
    container_name: broker2
    environment:
      KAFKA_ENABLE_KRAFT: yes
      KAFKA_CFG_PROCESS_ROLES: 'broker,controller'
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://broker2:9092
      KAFKA_CFG_BROKER_ID: 2
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: '1@broker1:9093,2@broker2:9093'
      ALLOW_PLAINTEXT_LISTENER: yes
      KAFKA_KRAFT_CLUSTER_ID: r4zt_wrqTRuT7W2NJsB_GA
    volumes:
      - broker2:/bitnami/kafka
  kafka-ui:
    container_name: kafka-ui
    image: 'provectuslabs/kafka-ui:latest'
    depends_on:
      - broker1
      - broker2
    ports:
      - "8092:8080"
    environment:
      KAFKA_CLUSTERS_0_BOOTSTRAP_SERVERS: 'broker1:9092,broker2:9092'
      KAFKA_CLUSTERS_0_NAME: local-kafka-cluster

Upvotes: 1

Adam
Adam

Reputation: 1939

This works as of Semptember 2023:

kafka:
  restart: always
  image: docker.io/bitnami/kafka:3.3
  volumes:
    - ./.volumes/kafka/data:/bitnami"
  ports:
    - 9092:9092
    - 9093:9093
  container_name: broker1
  environment:
    KAFKA_ENABLE_KRAFT: yes
    KAFKA_CFG_NODE_ID: 1
    KAFKA_CFG_PROCESS_ROLES: 'broker,controller'
    KAFKA_CFG_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
    KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
    KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
    KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://broker1:9092
    KAFKA_CFG_BROKER_ID: 1
    KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: '1@broker1:9093'
    ALLOW_PLAINTEXT_LISTENER: yes
    KAFKA_KRAFT_CLUSTER_ID: r4zt_wrqTRuT7W2NJsB_GA

Upvotes: 0

kolobok_ua
kolobok_ua

Reputation: 4190

So far the easiest way is to use confluent-local image. It uses default environment variables defined here.

---
version: '2'
services:
  broker:
    image: confluentinc/confluent-local:7.4.1
    ports:
      - "8082:8082"
      - "9092:9092"
      - "9101:9101"

Or if you need to use kafka-ui, you should also configure some env variables (for listeners etc):

---
version: '2'
services:

  broker:
    image: confluentinc/confluent-local:7.4.1
    hostname: broker
    container_name: broker
    ports:
      - "8082:8082"
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092'
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    ports:
      - 9999:8080
    environment:
      DYNAMIC_CONFIG_ENABLED: true
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:29092

Upvotes: 15

Dave
Dave

Reputation: 61

Confluent released a slimmed down confluent-local image in June 2023 that can run Kafka without ZooKeeper and can be used with Docker Compose.

If you just need Kafka, you can use the Confluent CLI instead of Docker Compose to manage the confluent-local container. One-liner from this quick start to get going: brew install confluentinc/tap/cli && confluent local kafka start

Disclaimer: I'm a Confluent employee

Upvotes: 6

MichaelCkr
MichaelCkr

Reputation: 690

Confluent published a working docker-compose.yaml without zookeeper in their repository cp-all-in-one.

There is a script (update_run.sh) used as a workaround

#!/bin/sh

# Docker workaround: Remove check for KAFKA_ZOOKEEPER_CONNECT parameter
sed -i '/KAFKA_ZOOKEEPER_CONNECT/d' /etc/confluent/docker/configure

# Docker workaround: Ignore cub zk-ready
sed -i 's/cub zk-ready/echo ignore zk-ready/' /etc/confluent/docker/ensure

# KRaft required step: Format the storage directory with a new cluster ID
echo "kafka-storage format --ignore-formatted -t $(kafka-storage random-uuid) -c /etc/kafka/kafka.properties" >> /etc/confluent/docker/ensure

which is called in the command of the docker-compose-setup

  broker:
    image: confluentinc/cp-kafka:7.2.x-latest
    hostname: broker
    container_name: broker
    ports:
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_NODE_ID: 1
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
    volumes:
      - ./update_run.sh:/tmp/update_run.sh
    command: "bash -c 'if [ ! -f /tmp/update_run.sh ]; then echo \"ERROR: Did you forget the update_run.sh file that came with this docker-compose.yml file?\" && exit 1 ; else /tmp/update_run.sh && /etc/confluent/docker/run ; fi'"

Upvotes: 8

daniil_
daniil_

Reputation: 980

According to "What’s New in Apache Kafka 3.3" document and "KIP-833: Mark KRaft as Production Ready" Kafka can work without Zookeeper (but there are some features yet works only by Apache ZooKeeper (ZK) mode).

Example (docker-compose.yml):

version: "2.5"
volumes:
  volume1:
services:
  kafka1:
    image: 'bitnami/kafka:3.3.1'
    container_name: kafka
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka1:9092
      - KAFKA_CFG_BROKER_ID=1
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka1:9093
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_KRAFT_CLUSTER_ID=r4zt_wrqTRuT7W2NJsB_GA
    volumes:
      - volume1:/bitnami/kafka
  kafka-ui:
    container_name: kafka-ui
    image: 'provectuslabs/kafka-ui:latest'
    ports:
      - "8080:8080"
    environment:
      - KAFKA_CLUSTERS_0_BOOTSTRAP_SERVERS=kafka1:9092
      - KAFKA_CLUSTERS_0_NAME=r4zt_wrqTRuT7W2NJsB_GA

You could try localhost:8080 and you will see that it works perfectly.

Upvotes: 12

Clear Love
Clear Love

Reputation: 71

You can use this image for no Zookeeper.

https://hub.docker.com/r/bitnami/kafka

Here is a example yaml.

version: "3"
services:
  kafka:
    image: 'bitnami/kafka:3.2.3'
    restart: "no"
    privileged: true
    ports:
      - 2181:2181
      - 19092:19092
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:19092,CONTROLLER://:2181
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:19092
      - KAFKA_BROKER_ID=1
      - [email protected]:2181
      - ALLOW_PLAINTEXT_LISTENER=yes

Upvotes: 7

Bash
Bash

Reputation: 197

Here's a Kafka Docker image which doesn't required Zookeeper (as described above):

https://hub.docker.com/r/bashj79/kafka-kraft

Disclaimer: I'm the author.

Upvotes: 18

Robin Moffatt
Robin Moffatt

Reputation: 32090

I read that Kafka no longer requires zookeeper

You may well have read that in the future Apache Kafka will not need Zookeeper - this is detailed in KIP-500

However, this is not yet implemented, so for the time being (January 2021) you will still need a Zookeeper in your Docker Compose ensemble.

Upvotes: 5

Related Questions