CODEWITHSUNDEEP
mysqlapache-kafka-connectdebezium
ForeverLearner
ForeverLearner

Reputation: 2113

Configure debezium connector to read mysql binary logs without any partial data

I am running debezium to capture data from mysql bin logs and publish on Kafka using kafka-connect. I got this error today:

[2019-06-18 11:32:38,175] ERROR Failed to properly convert data value for 'wallet_db_v3.user_registrations.msisdn' of type VARCHAR for row [824, 2, 2019-06-18T06:20:26]: (io.debezium.relational.TableSchemaBuilder:253)
java.lang.IllegalArgumentException: Unexpected value for JDBC type 12 and column msisdn VARCHAR(50) CHARSET utf8 NOT NULL: class=class java.lang.Integer
at io.debezium.jdbc.JdbcValueConverters.handleUnknownData(JdbcValueConverters.java:1164)
at io.debezium.jdbc.JdbcValueConverters.convertValue(JdbcValueConverters.java:1207)
at io.debezium.connector.mysql.MySqlValueConverters.convertString(MySqlValueConverters.java:380)
at io.debezium.connector.mysql.MySqlValueConverters.lambda$converter$13(MySqlValueConverters.java:293)
at io.debezium.relational.TableSchemaBuilder.lambda$createValueGenerator$2(TableSchemaBuilder.java:249)
at io.debezium.relational.TableSchema.valueFromColumnData(TableSchema.java:145)
at io.debezium.connector.mysql.RecordMakers$1.insert(RecordMakers.java:244)
at io.debezium.connector.mysql.RecordMakers$RecordsForTable.create(RecordMakers.java:456)
at io.debezium.connector.mysql.BinlogReader.handleInsert(BinlogReader.java:833)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:501)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1095)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:943)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)
at java.lang.Thread.run(Thread.java:748)

The table structure :

'id', 'int(10) unsigned', 'NO', 'PRI', NULL, 'auto_increment'
'msisdn', 'varchar(50)', 'NO', '', NULL, ''
'device_id', 'varchar(50)', 'NO', '', NULL, ''
'status', 'enum(\'waiting_for_validation\',\'approved\',\'rejected\',\'expired\')', 'NO', '', NULL, ''
'rejection_reason', 'enum(\'max_otp_validations_reached\',\'canceled_by_user\')', 'YES', '', NULL, ''
'updated_at', 'datetime', 'YES', '', 'CURRENT_TIMESTAMP', 'on update CURRENT_TIMESTAMP'
'created_at', 'datetime', 'NO', '', NULL, ''
'channel', 'varchar(50)', 'YES', '', NULL, ''

The full row should look like:

select * from user_registrations where id = 824;
['824', '+115004937263', '56DA9A2FD4D91320D01514ED497C719E5DC9323F', 'approved', NULL, '2019-06-18 06:20:26', '2019-06-18 06:20:23', NULL]

If the table is queried now, this row appears complete.

I doubt that debezium connector is reading the records before it has been written completely into the mysql binlog. Has anyone configured debezium in a way that ensures it reads after a record has been completely written into the binlog

The my.cnf settings for reference:

log-bin=/var/log/mysql/mysql-bin
binlog_format=row
binlog_row_image=full

Upvotes: 0

Views: 1453

Answers (1)

Jiri Pechanec
Jiri Pechanec

Reputation: 1976

The problem is with source table - it is NDB table that is not controlled via binlog_row_image parameter set to full.

Upvotes: 1

Related Questions