CODEWITHSUNDEEP
databaseapache-kafkastreamdebezium
Roman
Roman

Reputation: 11

Debezium connector with column.include.list

How do I combine table.include.list and column.include.list in Debezium connector configuration if I need to snapshot part of one table and full data from another table?

Example connector configuration:

"table.include.list":"schema.table1, schema.table2",
"column.include.list":"schema.table1.col1, schema.table1.col2"

With this I get all columns of table1 in kafka.

Only managed how to do it using different connectors.

I want to get all columns from table2 and 2 columns from table1 using one connector, is it possible?

Upvotes: 1

Views: 6173

Answers (2)

Anton Ermakov
Anton Ermakov

Reputation: 469

From Debezium documentation for column.include.list configuration attribute:

An optional, comma-separated list of regular expressions that match the fully-qualified names of columns to include in change event record values. Fully-qualified names for columns are of the form databaseName.tableName.columnName.

So you have to configure 2 things:

  1. fully qualified table name in the table_include_list attribure (which is also a comma-separated list of regular expressions)
  2. fully qualified column names in the column_include_list attribure

If you want to include all columns from table, then configuration of column names in the column_include_list attribute is not necessary.

For example:

{
    ...
    "table.include.list": "schema\\.table1,schema\\.table2",
    "column.include.list": "schema\\.table1\\.(col1|col2)"
    ...
}

Column exclusion is configured in the same way, but using the column_exclude_list attribute.

See details here (i.e. for MySQL connector, other connectors are similar): https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-table-include-list

Upvotes: 1

Gunnar
Gunnar

Reputation: 19010

This should work:

"table.include.list":"schema\.table1,schema\.table2",
"column.include.list":"schema\.table1\.{col1|col2},schema\.table2\..*"

Upvotes: 2

Related Questions