How to implement change data capture (CDC) using Kafka Connect for Sybase ASE database?

Question

I have a requirement where I have to stream live data base updates into KAFKA topics. So basically, whenever something is added, updated or deleted in the database I want that update to be pushed in kafka topic. I saw some articles on how we can do that for mysql and postgresql using debezium. But the database I want to monitor is Sybase ASE. Does debezium support sybase ASE?

If yes, can someone give me good documentation on it? I could not find any on the internet.

If no, what other way can I implement this change data capture (CDC) functionality for sybase ASE database?

Answer 1

NOTE: While I've got a good bit of experience with the Sybase(SAP) ASE and Repserver products, I have very little experience with SQL Server and no experience with debezium, so fwiw ...

According to debezium documentation: connectors, Sybase (SAP) ASE is not supported.

While ASE and (MS)SQL Server have a common history, it does not appear the 2x products provide the same log-reading capability (not surprising since the 2x products went their separate way about the time the Sybase Repserver product became available); according to the debezium documentation:

• SQL Server provides CDC 'tables' that debezium reads from; jumping over to the SQL Server CDC explanation it looks like the log is read (under the covers) and the DML operations are dumped into these CDC 'tables'
• ASE has an undocumented API for reading the log; this API converts the log entries into Log Transfer Language (LTL) records which are then passed to a (Sybase/SAP) Repserver (or other piece of software capable of mimicking a Repserver)

Short of contacting the debezium support team and requesting they add support for Sybase (SAP) ASE, it seems (to me) that you may need to look for another product.

---

While it might be a 'fun' little project to see if you can emulate SQL Server's CDC tables, that would be a major undertaking, eg,

• you'd be looking at having to add triggers to all (ASE) tables of interest
• the triggers would need to copy most (all?) DML operations into new 'CDC' tables
• the volumes on those 'CDC' tables could get quite large (compared to the based table's size), which ...
• would require a good bit of infrastructure to manage (and prune) said 'CDC' tables
• you'd need to figure out how to emulate the procs SQL Server provides for managing the 'CDC' tables
• and even if you got this far there's the question of whether or not you could get debezium to connect to ASE

Another idea re: rolling your own solution ...

The Sybase(SAP) ASE SDK (upon which the ASE and Repserver products are built) could be used to design a module to connect from Repserver to Kafka, but that's a whole 'nother (lengthy) discussion better suited for someone who's into SDK programming ...

---

As for 'other products' ... Mr Google is likely a good start at this point.

The only product I'm familiar with that makes use of the (undocumented) API for reading from ASE's log, and which claims to interact with Kafka, is the Qlik Replicate product.

Answer 2

Ideally CDC should be done by reading from the DB log (viz. redo log of Oracle, binlog of MySQL, SSTable of Cassandra, etc.).

For the proprietary DBs, usually the vendor provides an API or tool to read the log. For e.g. Oracle provides Golden Gate. I haven't seen anything similar from Sybase.

One of the possible options with Sybase is JDBC. If you're using Kafka Connect, you can have a look at the JDBC Connector. However, I think it may not be a good solution, if the data volume is significant.

Another alternative could be that the process that writes to the DB also produces a message to the Kafka topic. That is not CDC in true sense, but it can be a workable solution. But you need to handle scenarios where writing to DB is successful but message production to Kafka fails.