Sumalah
Reputation: 13
Debezium throws an error when connecting to RDS Aurora instance
Here are elements of my infrastructure related to this problem:
- RDS Aurora, which runs on MySql 5.7 with binlogs enabled in row-based mode
- AWS MSK which runs Kafka cluster (I don't know if that's important, but it runs on version 2.7.0)
- EC2 instance with Kafka-connect (as a Docker image) where runs a source connector which uses Debezium
I'm trying to use Debezium to get all changes from the Aurora instance and put them to Kafka but unfortunately, it is not working. Here is the connector configuration I'm using:
{
"name": "aurora-connector-test",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"tasks.max": "1",
"database.hostname": "URL",
"database.port": "3306",
"database.user": "debezium",
"database.password": "PASSWORD",
"database.server.id": "129056",
"database.server.name": "aurora-connector-db-test",
"database.allowPublicKeyRetrieval":"true",
"database.include.list": "DATABASE",
"database.history.kafka.bootstrap.servers": "BROKER1:9092,BROKER2:9092,BROKER3:9092",
"database.history.kafka.topic": "schema-changes.DATABASE",
"transforms": "route",
"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": "([^.]+)\\.([^.]+)\\.([^.]+)",
"transforms.route.replacement": "$3"
}
}
After creating a connector using the above configuration I can see that Debezium is able to connect to the Aurora database because it's listing all available tables in the logs. Sadly, at some point, I'm getting Communications link failure error. Here is a chunk of logs:
connect_1 | 2021-01-12T08:47:39.106265621Z 2021-01-12 08:47:39,106 INFO MySQL|aurora-connector-db-test|snapshot Step 5: committing transaction [io.debezium.connector.mysql.SnapshotReader]
connect_1 | 2021-01-12T08:47:39.107645625Z 2021-01-12 08:47:39,107 ERROR MySQL|aurora-connector-db-test|snapshot Failed due to error: Aborting snapshot due to error when last running 'SHOW FULL TABLES IN `sys` where Table_Type = "BASE TABLE"': Communications link failure
connect_1 | 2021-01-12T08:47:39.113018742Z org.apache.kafka.connect.errors.ConnectException: Communications link failure
connect_1 | 2021-01-12T08:47:39.113022120Z
connect_1 | 2021-01-12T08:47:39.113025636Z The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. Error code: 0; SQLSTATE: 08S01.
connect_1 | 2021-01-12T08:47:39.113029640Z at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
connect_1 | 2021-01-12T08:47:39.113033236Z at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:207)
connect_1 | 2021-01-12T08:47:39.113036512Z at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:847)
connect_1 | 2021-01-12T08:47:39.113039728Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
connect_1 | 2021-01-12T08:47:39.113055546Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
connect_1 | 2021-01-12T08:47:39.113059492Z at java.base/java.lang.Thread.run(Thread.java:834)
connect_1 | 2021-01-12T08:47:39.113062958Z Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
connect_1 | 2021-01-12T08:47:39.113066762Z
connect_1 | 2021-01-12T08:47:39.113070570Z The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
connect_1 | 2021-01-12T08:47:39.113074098Z at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
connect_1 | 2021-01-12T08:47:39.113077538Z at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
connect_1 | 2021-01-12T08:47:39.113080901Z at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:835)
connect_1 | 2021-01-12T08:47:39.113084338Z at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:455)
connect_1 | 2021-01-12T08:47:39.113087583Z at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:240)
connect_1 | 2021-01-12T08:47:39.113090489Z at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:199)
connect_1 | 2021-01-12T08:47:39.113093761Z at io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory$1(JdbcConnection.java:191)
connect_1 | 2021-01-12T08:47:39.113096988Z at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:789)
connect_1 | 2021-01-12T08:47:39.113100397Z at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:784)
connect_1 | 2021-01-12T08:47:39.113103714Z at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:748)
connect_1 | 2021-01-12T08:47:39.113107314Z ... 3 more
connect_1 | 2021-01-12T08:47:39.113110461Z Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
connect_1 | 2021-01-12T08:47:39.113113684Z
connect_1 | 2021-01-12T08:47:39.113117610Z The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
connect_1 | 2021-01-12T08:47:39.113121639Z at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
connect_1 | 2021-01-12T08:47:39.113125289Z at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
connect_1 | 2021-01-12T08:47:39.113128514Z at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
connect_1 | 2021-01-12T08:47:39.113131552Z at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
connect_1 | 2021-01-12T08:47:39.113134638Z at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
connect_1 | 2021-01-12T08:47:39.113137876Z at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
connect_1 | 2021-01-12T08:47:39.113140913Z at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
connect_1 | 2021-01-12T08:47:39.113148186Z at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
connect_1 | 2021-01-12T08:47:39.113151641Z at com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:91)
connect_1 | 2021-01-12T08:47:39.113155079Z at com.mysql.cj.NativeSession.connect(NativeSession.java:152)
connect_1 | 2021-01-12T08:47:39.113158553Z at com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:955)
connect_1 | 2021-01-12T08:47:39.113162110Z at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:825)
connect_1 | 2021-01-12T08:47:39.113165344Z ... 10 more
connect_1 | 2021-01-12T08:47:39.113168317Z Caused by: java.net.ConnectException: Connection refused (Connection refused)
connect_1 | 2021-01-12T08:47:39.113171590Z at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
connect_1 | 2021-01-12T08:47:39.113174792Z at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
connect_1 | 2021-01-12T08:47:39.113178326Z at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
connect_1 | 2021-01-12T08:47:39.113181678Z at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
connect_1 | 2021-01-12T08:47:39.113185328Z at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
connect_1 | 2021-01-12T08:47:39.113188827Z at java.base/java.net.Socket.connect(Socket.java:609)
connect_1 | 2021-01-12T08:47:39.113192354Z at com.mysql.cj.protocol.StandardSocketFactory.connect(StandardSocketFactory.java:155)
connect_1 | 2021-01-12T08:47:39.113195838Z at com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:65)
connect_1 | 2021-01-12T08:47:39.113199398Z ... 13 more
connect_1 | 2021-01-12T08:47:39.113249277Z 2021-01-12 08:47:39,113 ERROR || WorkerSourceTask{id=aurora-connector-test-0} Task is being killed and will not recover until manually restarted [org.apache.kafka.connect.runtime.WorkerTask]
connect_1 | 2021-01-12T08:47:39.113256190Z 2021-01-12 08:47:39,113 INFO MySQL|aurora-connector-db-test|task Stopping MySQL connector task [io.debezium.connector.mysql.MySqlConnectorTask]
connect_1 | 2021-01-12T08:47:39.113259635Z 2021-01-12 08:47:39,113 INFO MySQL|aurora-connector-db-test|task ChainedReader: Stopping the snapshot reader [io.debezium.connector.mysql.ChainedReader]
connect_1 | 2021-01-12T08:47:39.113263070Z 2021-01-12 08:47:39,113 INFO MySQL|aurora-connector-db-test|task Discarding 0 unsent record(s) due to the connector shutting down [io.debezium.connector.mysql.SnapshotReader]
connect_1 | 2021-01-12T08:47:39.113266516Z 2021-01-12 08:47:39,113 INFO MySQL|aurora-connector-db-test|task Discarding 0 unsent record(s) due to the connector shutting down [io.debezium.connector.mysql.SnapshotReader]
connect_1 | 2021-01-12T08:47:39.113420628Z 2021-01-12 08:47:39,113 INFO MySQL|aurora-connector-db-test|task [Producer clientId=aurora-connector-test-dbhistory] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. [org.apache.kafka.clients.producer.KafkaProducer]
connect_1 | 2021-01-12T08:47:39.114679266Z 2021-01-12 08:47:39,114 INFO MySQL|aurora-connector-db-test|task Connector task finished all work and is now shutdown [io.debezium.connector.mysql.MySqlConnectorTask]
connect_1 | 2021-01-12T08:47:39.114799622Z 2021-01-12 08:47:39,114 INFO || [Producer clientId=connector-producer-aurora-connector-test-0] Closing the Kafka producer with timeoutMillis = 30000 ms. [org.apache.kafka.clients.producer.KafkaProducer]
I know that my security groups on AWS are configured correctly because I'm able to connect to the Aurora instance remotely from the EC2 instance. Additionally, I was using a very similar connector configuration for MySQL instead of Aurora on the same EC2 instance and it was able to create messages in the same Kafka - hence I think that the problem is with Aurora configuration (I'm using the default configuration with row-based binlogs enabled), but maybe I'm wrong.
Do you guys have any ideas on that issue?
Upvotes: 0
Views: 1168
Answers (1)
user5145
Reputation: 21
The debezium user needs these privileges: SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES. My only guess is to check if the user has all of them.
Upvotes: 1
Related Questions
- Mysql debezium connector for rds in production caused deadlocks
- what would cause this debezium kafka connector error?
- Using Debezium MySQL connector with MSK shows "INVALID_REPLICATION_FACTOR"
- Error with using Debezium Connector for Kafka and MySQL
- Debezium Connector for RDS Aurora
- Debezium kafka connect connector not updating successfully
- Getting error while connecting to Amazon Aurora with MySQL Workbench
- Debezium error when connecting to kafka multi-broker in docker swarm
- AWS Aurora - Unable to connect to aws aurora db
- Stuck on talking to RDS/Aurora from node