CODEWITHSUNDEEP
dockerapache-kafkaapache-kafka-connectconfluent-platform
Jeff Beagley
Jeff Beagley

Reputation: 1025

Custom Environment Variables for Kafka Connect via Docker

Is there a way to provide custom variables via Docker-Compose that can be referenced within a Kafka Connector config?

I have the following setup in my docker-compose.yml:

- "sql_server=1.2.3.4"
- "sql_database=db_name"
- "sql_username=some_user"
- "sql_password=nahman"
- "sql_applicationname=kafka_connect"

Here is my .json configuration file:

{
  "name": "vwInv_Tran_Amounts",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "src.consumer.interceptor.classes": "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor",
    "tasks.max": 2,
    "connection.url": "jdbc:sqlserver://${sql_server};database=${sql_database};user=${sql_username};password={sql_password};applicationname={sql_applicationname}",
    "query": "SELECT * FROM vwInv_Tran_Amounts",
    "mode": "timestamp",
    "topic.prefix": "inv_tran_amounts",
    "timestamp.column.name": "timestamp",
    "incrementing.column.name": "Inv_Tran_ID"
  }
}

I was able to reference the environment variables using this method with Elastic Logstash, but it doesn't appear to work here.

Whenever loading it via curl I receive:

The connection string contains a badly formed name or value. for configuration Couldn't open connection to jdbc:sqlserver://${sql_server};database=${sql_database};user=${sql_username};password={sql_password};applicationname={sql_applicationname}\nInvalid value com.microsoft.sqlserver.jdbc.SQLServerException: The connection string contains a badly formed name or value.

EDIT///////// I tried prefixing environment varibles like CONNECT_SQL_SERVER and that didn't work.

Upvotes: 0

Views: 3672

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191743

I feel like you are looking for Externalizing Kafka Connect secrets, but that would require mounting a file, not using env vars.

JSON Connector config files aren't loaded on Docker container startup. I made this issue to see if this would be possible.

You would have to template out the JSON file externally, then HTTP-POST them to the port exposed by the container.

Tried prefixing environment varibles like CONNECT_SQL_SERVER

Those values would go into the Kafka Connect Worker properties, not the properties that need to be loaded by a specific connector task.

Upvotes: 1

Related Questions