CODEWITHSUNDEEP
mysqldebezium
M4rk
M4rk

Reputation: 2272

Is it possible to merge two Debezium connectors?

Dealing with large MySQL DBs (10s TB), I find myself having to split up a connector so I can process more rows at a time (every single connector can process one table a time).

Once the initial sync is complete, and it switches to incremental, what is the cleanest way of merging the two connectors?

Is it even possible?

Upvotes: 0

Views: 367

Answers (1)

Naros
Naros

Reputation: 21113

Since both connectors are created with different database.server.name values, their associated topics are likely prefixed with different values and so attempting to merge multiple connectors would be highly cumbersome and error-prone.

What I would suggest is if you have a large volume of data that you need to snapshot, don't rely on the initial snapshot phase to capture the data. Instead, configure a single connector to use the schema_only snapshot mode so that the schemas get captured before streaming starts. Then you can leverage incremenal snapshots that run in parallel with streaming to capture the 10TB of data concurrently.

Upvotes: 1

Related Questions