Anton
Anton

Reputation: 591

Kafka JDBC Source connector not respecting search path

We've setup our postgres source database so that we have all tables that we are interested in, in two schemas. Essentially we have schema1 and schema2, and the tables in these schemas are identical. When we make a deployment/migration we get rid of schema1 and create schema3, always having two active schemas, one of them being a 'backup' in case we need to revert.

This is typically fine because we've defined a search path so that when running unqualified queries (without schema name, just select * from table_name) it always picks the correct schema. For all (most?) clients this works, as we just need to write queries without the schema and everything is set.

For the JDBC source connector this isn't the case though, and it cannot handle such a case, where a single whitelisted table lives in two schemas. I've looked at various workarounds but not found anything that works, I also don't have much control over the postgres database so making drastic changes to how we operate there isn't feasible.

One workaround which I identified is just using the jdbc connector in the query mode, but using this implementation we lose much of the convenience that the connector gives. It also makes the implementation unnecessarily complex.

Other than that I can't find a way to make this connector work with our setup.

Upvotes: 0

Views: 128

Answers (0)

Related Questions