CODEWITHSUNDEEP
dockerapache-kafkadocker-compose
Leo Jiang
Leo Jiang

Reputation: 26153

Kafka Broker running in Docker is repeatedly restarting?

In my dev environment, Kafka Broker usually runs fine. However, every couple of days, it gets stuck in an endless restart cycle. It would restart every ~10min and no services can connect to it. The only way for me to fix it is docker compose down && docker compose up. The last few log lines before restarting are a bunch of warnings like:

WARN [Producer clientId=_confluent_balancer_api_state-producer-1]
  Connection to node 1 (broker/172.18.0.5:29092) could not be established.
  Broker may not be available. (org.apache.kafka.clients.NetworkClient)
WARN [Producer clientId=ConfluentBalancerSampleStoreProducer]
  Connection to node 1 (broker/172.18.0.5:29092) could not be established.
  Broker may not be available. (org.apache.kafka.clients.NetworkClient)

There's also this:

EndOfStreamException: Unable to read additional data from server sessionid 0x1000003dd170035, likely server has closed socket        
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)

A potentially relevant error that occurs earlier in the logs is:

ERROR Skip updating configuration of topic _confluent_balancer_partition_samples due to exception. (com.linkedin.kafka.cruisecontrol.SbkTopicUtils)
java.lang.RuntimeException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: 
  The topic '_confluent_balancer_partition_samples' does not exist.

Here's the Docker Compose service (mostly copied from Confluent):

  broker:
    image: confluentinc/cp-server:7.0.1
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - 127.0.0.1:9092:9092
      - 127.0.0.1:9101:9101
    restart: always
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://127.0.0.1:9092
      # KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
      KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      KAFKA_LOG4J_LOGGERS: org.apache.zookeeper=WARN,org.apache.kafka=WARN,kafka=WARN,kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN
      KAFKA_LOG4J_ROOT_LOGLEVEL: WARN
      KAFKA_TOOLS_LOG4J_LOGLEVEL: WARN
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'false'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'
      CONFLUENT_REPORTERS_TELEMETRY_AUTO_ENABLE: 'false'

I'm fairly new to Kafka/Docker, I don't know what most of these settings mean. Others have suggested changing KAFKA_ADVERTISED_LISTENERS and KAFKA_LISTENERS, but it didn't work. What are some others things I could try?

Upvotes: 2

Views: 1781

Answers (1)

Luis Estrada
Luis Estrada

Reputation: 449

If you want to connect outside the localhost (127.0.0.1), the in the broker service set the ports to:

ports:
  - "9092:9092"
  - "9101:9101"

And assuming Kakfka is on server ip 172.18.0.5 (from the logs)

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://172.18.0.5:9092

Upvotes: 1

Related Questions