CODEWITHSUNDEEP
jdbcapache-kafkaapache-kafka-connect
oalmora90
oalmora90

Reputation: 13

Kafka JDBC Source connector: create topics from column values

I have a microservice that uses OracleDB to publish the system changes in the EVENT_STORE table. The table EVENT_STORE contains a column TYPE with the name of the type of the event.

It is possible that JDBC Source Kafka Connect take the EVENT_STORE table changes and publish them with the value of column TYPE in the KAFKA-TOPIC?

It is my source kafka connector config:

{
  "name": "kafka-connector-source-ms-name",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "tasks.max": "1",
    "connection.url": "jdbc:oracle:thin:@localhost:1521:xe",
    "connection.user": "squeme-name",
    "connection.password": "password",
    "topic.prefix": "",
    "table.whitelist": "EVENT_STORE",
    "mode": "timestamp+incrementing",
    "timestamp.column.name": "CREATE_AT",
    "incrementing.column.name": "ID",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "config.action.reload": "restart",
    "errors.retry.timeout": "0",
    "errors.retry.delay.max.ms": "60000",
    "errors.tolerance": "none",
    "errors.log.enable": "false",
    "errors.log.include.messages": "false",
    "connection.attempts": "3",
    "connection.backoff.ms": "10000",
    "numeric.precision.mapping": "false",
    "validate.non.null": "true",
    "quote.sql.identifiers": "ALWAYS",
    "table.types": "TABLE",
    "poll.interval.ms": "5000",
    "batch.max.rows": "100",
    "table.poll.interval.ms": "60000",
    "timestamp.delay.interval.ms": "0",
    "db.timezone": "UTC"
  }
}

Upvotes: 1

Views: 1127

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191725

You can try the ExtractTopic transform to pull a topic name from a field

Add the following properties to the JSON

transforms=ValueFieldExample
transforms.ValueFieldExample.type=io.confluent.connect.transforms.ExtractTopic$Value
transforms.ValueFieldExample.field=TYPE

Upvotes: 1

Related Questions