How to configure bitnami ksqldb docker image in a docker-compose

Question

I honestly never used an image from bitnami and I am having difficulties in understanding how to use the KSQLDB one. I am pretty sure I am doing something pretty stupid since I misunderstood something quite basic.

This is my basic docker-compose:

services:
  ksqldb-server:
    image: bitnami/ksql:latest
    hostname: ksqldb-server
    container_name: ksqldb-server
    ports:
      - "8088:8088"
    volumes:
      - "./ksql-server.properties:/opt/bitnami/ksql/config/ksql-server.properties"
    environment:
      # KSQL
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: my-remote-server:9092
      KSQL_MOUNTED_CONF_DIR: "/opt/bitnami/ksql/config"
  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.29.0
    container_name: ksqldb-cli
    depends_on:
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

My goal was to set these few ENVS in the BITNAMI documentation and then set the others through conf file. I want to keep default values for all the others.

Here is my config file:

security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="usernameXXX" password="passwordXXX";
ssl.endpoint.identification.algorithm=HTTPS
ksql.internal.topic.replicas=3
ksql.sink.replicas=3
ksql.streams.replication.factor=3
ksql.logging.processing.topic.replication.factor=3
ksql.logging.processing.stream.auto.create=true
ksql.logging.processing.topic.auto.create=true
ksql.log4j.root.loglevel=DEBUG

Since bitname uses user 1001 I do chown -R 1001:1001 ksql-server.properties.

When I launch this I get:

ksqldb-server  | ksql 09:04:32.27 INFO  ==> 
ksqldb-server  | ksql 09:04:32.28 INFO  ==> ** Starting KSQL setup **
ksqldb-server  | ksql 09:04:32.30 INFO  ==> Validating settings in KSQL_* env vars
ksqldb-server  | ksql 09:04:32.33 INFO  ==> Initializing Confluent KSQL
ksqldb-server  | ksql 09:04:32.34 INFO  ==> Injected configuration file found. Skipping default configuration
ksqldb-server  | ksql 09:04:32.34 INFO  ==> Waiting for Kafka brokers to be up
ksqldb-server exited with code 1

I also tried to remove: KSQL_MOUNTED_CONF_DIR: "/opt/bitnami/ksql/config"

and set:

volumes:
      - "./ksql-server.properties:/opt/bitnami/ksql/etc/ksqldb/ksql-server.properties"

Nothing changed, still detecting injected config, no debug logs.

If I remove the volume at all (no config file) I get a "no config detected using default" and then an obvious error on Kafka endpoint.

My initial guess would have been that the /opt/bitnami/ksql/etc/ksqldb folder is used for creating the default file when you don't provide it with KSQL_MOUNTED_CONF_DIR. Nevertheless, even substituting the file in that folder creates an "injected configuration file found" log, so I have no idea how this works.

Moreover, looking in that folder with docker run -it bitnami/ksql /bin/bash I saw lots of templates. In particular one is called ksqldb-server.properties.template and has inside:

{% set kr_props = env_to_props('KSQL_', '') -%}
{% for name, value in kr_props.items() -%}
{{name}}={{value}}
{% endfor -%}

That seems the approach used in the docker image confluentinc/ksqldb-server and that would be my favourite way to configure. I tried (without hopes to add other ENVS not written in the bitnami's very small docs like this:

      KSQL_SASL_JAAS_CONFIG: >
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="usernameXXX"
        password="passwordXXX";
      KSQL_SECURITY_PROTOCOL: SASL_SSL
      KSQL_SASL_MECHANISM: PLAIN
      KSQL_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: "HTTPS"

Same error.

In short:

How does this image work?
how to configure it?
is it possible to set it so that I can use ENVS like in confluentinc/ksqldb-server?

How to configure bitnami ksqldb docker image in a docker-compose

Answers (1)

Related Questions