CODEWITHSUNDEEP
dockerapache-kafkadocker-composeconfluent-platform
Jon Bates
Jon Bates

Reputation: 3183

Speeding up Confluent docker load time

I have a docker-compose file stat spins up a single-node Kafka, Zookeeper & schema registry stack for testing my application. Currently, it takes a couple of minutes to get the stack available; are there any settings to speed up the launch time?

The config I'm using (apart from SSL) is as follows:

kafka:
  image: confluentinc/cp-kafka:3.3.1
  depends_on:
    - zookeeper
  hostname: kafka
  ports:
    - 9092:9092
  environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_ADVERTISED_LISTENERS: SSL://kafka:9092
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
    KAFKA_LOG4J_ROOT_LOGLEVEL: WARN
    KAFKA_JMX_PORT: 9585
    KAFKA_JMX_HOSTNAME: kafka
    KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=9585"
  volumes:
    - ../../txs-data/kafka-data:/var/lib/kafka/data

Upvotes: 1

Views: 692

Answers (2)

OneCricketeer
OneCricketeer

Reputation: 192033

Other than using a newer container version, there isn't a way to make it faster.

Zookeeper starts fairly quickly, but Kafka relies on Zookeeper and needs to coordinate extra tasks to elect a leader, load some other metadata, etc. (Update: No longer needed)

If you're adding the schema registry on top of that, it requires Kafka to start, then create it's _schemas topic, which requires a round-trip to Zookeeper.

All-in-all, there is a lot of pre-initialization steps happening, of which all are required and cannot be skipped to reduce the start-time.

Assuming you're running this as part of a JVM testing framework, the "faster" way would be to use embedded versions of each of the services.

Upvotes: 1

Gilles Philippart
Gilles Philippart

Reputation: 645

You can now start Kafka without Zookeeper:

git clone https://github.com/confluentinc/cp-all-in-one
cd cp-all-in-one/cp-all-in-one-kraft
docker compose up broker

Upvotes: 0

Related Questions