CODEWITHSUNDEEP
postgresqlapache-kafkaapache-kafka-connectdebezium
user1578872
user1578872

Reputation: 9038

Kafka source connector for postgres - Day0 load

I am looking for a Kafka source connector for Day0 load from Postgres to Kafka.

Came across Debezium postgres connector.

Docker image,

debezium/connect:1.4

docker run -it --rm --name postgres-connect -p 8083:8083 -e BOOTSTRAP_SERVERS=host1:8080 -e GROUP_ID=test-1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses debezium/connect:1.4

How to pass the postgres host details and kafka sasl config?

Any help would be appreciated.

Upvotes: 1

Views: 715

Answers (1)

Iskuskov Alexander
Iskuskov Alexander

Reputation: 4375

1. SASL configuration

1.1. In common case you need to add the following properties to your connect-distributed.properties:

sasl.mechanism=PLAIN
security.protocol=SASL_PLAINTEXT

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="connect" \
  password="connect-secret";

producer.sasl.mechanism=PLAIN
producer.security.protocol=SASL_PLAINTEXT
producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="connect" \
  password="connect-secret";

consumer.sasl.mechanism=PLAIN
consumer.security.protocol=SASL_PLAINTEXT
consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="connect" \
  password="connect-secret";

Source: StackOverflow answer "ACL configuration in Kafka connect is not working"

Reference: Kafka Connect Security docs

1.2. For debezium/connect docker image you can try to pass SASL config directly via environment variables (using these transformation steps):

docker run -it --rm --name postgres-connect -p 8083:8083 \
  -e BOOTSTRAP_SERVERS=host1:8080 -e GROUP_ID=test-1 \
  -e CONFIG_STORAGE_TOPIC=my_connect_configs \
  -e OFFSET_STORAGE_TOPIC=my_connect_offsets \
  -e STATUS_STORAGE_TOPIC=my_connect_statuses \
  -e CONNECT_SASL_MECHANISM=PLAIN \
  -e CONNECT_SECURITY_PROTOCOL=SASL_PLAINTEXT \
  -e CONNECT_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="connect" password="connect-secret"; \
  -e CONNECT_PRODUCER_SASL_MECHANISM=PLAIN \
  -e CONNECT_PRODUCER_SECURITY_PROTOCOL=SASL_PLAINTEXT \
  -e CONNECT_PRODUCER_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="connect" password="connect-secret"; \
  -e CONNECT_CONSUMER_SASL_MECHANISM=PLAIN \
  -e CONNECT_CONSUMER_SECURITY_PROTOCOL=SASL_PLAINTEXT \
  -e CONNECT_CONSUMER_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="connect" password="connect-secret"; \
debezium/connect:1.4

2. PostgreSQL host configaration

Host details should be passed via Kafka Connect REST API using connector config:

curl -i -X PUT -H "Content-Type:application/json" \
    http://localhost:8083/connectors/debezium_postgres_source/config \
    -d '{
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "tasks.max": "1",
    "database.hostname": "source-db",
    "database.port": "5432",
    "database.user": "postgresusersource",
    "database.password": "postgrespw",
    "database.dbname" : "sourcedb",
    "database.server.name": "dbserver1"
}'

Upvotes: 2

Related Questions