CODEWITHSUNDEEP
javapostgresqlapache-kafkaapache-kafka-connectconfluent-platform
Faheem Sultan
Faheem Sultan

Reputation: 283

kafka JDBC source connector not able to pick up postgress table

i have a standalone kafka runing on windows and a JDBC source connector connected with postgress db which is available on my machine. when i try to run the connector it shows a warning that no tasks will be run because no tables were found. whereas if i connect the same connector with mysql db (which is also locally installed). it is able to pick up the tables and works fine. please note that i am using confluent's jdbc connector with default apache-kafka.

below is the warning i get with postgress enter image description here

here are the configurations of my jdbc source worker

name=postgress-to-topic
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
connection.url=jdbc:postgresql://localhost:5432/test-kafka
connection.user=postgres
connection.password=postgress
topic.prefix=postgres-
#mode=bulk
mode=incrementing
incrementing.column.name=id
errors.log.enable=true
errors.log.include.messages=true
tasks.max=1
poll.interval.ms =100
schema.pattern=test-kafka
#catalog.pattern=test-kafka
table.whitelist=mytable
#dialect.name=PostgreSqlDatabaseDialect
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
topic.prefix=postgress-

these are mysql configs in the same file which are commented for now.

   #mysql configs
    #connection.user=root
    #connection.password=root
    #connection.url=jdbc:mysql://localhost:3306/test-kafka?useSSL=false
    #name=mysql-to-topic
    #topic.prefix=mysql2-
    #schema.pattern=test-kafka
    #table.whitelist=user-data-1

below is how my postgress db looks like

enter image description here

and below is how my mysql db looks like (which is working fine)

enter image description here

i don't have much experience with postgress and not able to understand why it's not able to pick up table in postgress.

Upvotes: 0

Views: 1428

Answers (2)

Faheem Sultan
Faheem Sultan

Reputation: 283

database and schema are two different things in postgress, heirarchy is something like below

database-> {schema1, schema2}
             schema1-> {table, table}
             schema2-> {table, table}

i was giving the schema.pattern as db name which was causing the issue, once i gave proper schema name it worked fine.

Upvotes: 0

Peter
Peter

Reputation: 11

Believe the confluent connector connector is the debezium connector

https://www.confluent.io/hub/debezium/debezium-connector-postgresql

which has documentation here

https://debezium.io/documentation/reference/connectors/postgresql.html

this needs to use logical replication from Postgres in order to pick up changes. You need to configure the system to replicate the tables that you want to pick up changes from and create a publication.

See:

https://www.postgresql.org/docs/10/logical-replication.html

Upvotes: 1

Related Questions