CODEWITHSUNDEEP
sqldesign-patternsapache-kafkaspring-kafka
Sarthak Sahu
Sarthak Sahu

Reputation: 57

Importing data into database from Kafka

I have a general question about the "ethics" of integrating messages into database via a Kafka consumer.

I always recommend that always keep a separate consumer for each topic as you would have different processing logic for each type of message. So a Spring based Kafka consumer with a Crud repository to do repo.save().

But my colleagues, who are SQL veterans, want such consumers to be generic and rather have the processing logic run at the DB level (stored procedures ofc). So all the consumers do is poll the topic and put it into a staging table and then the integration is taken care by the sp.

Now why I do not want this to be the case is because the repo.save() helps to maintain a good ORM between the message and the DB table, and any unwarranted changes at either end would be somewhat controllable. Whereas the concept of sp gives you more control with handing the data as you wish and could lead to some table structure changes.

Could anybody outline what would be a good design practice for this?

Upvotes: 0

Views: 1659

Answers (1)

JavaTechnical
JavaTechnical

Reputation: 9347

But my colleagues, who are SQL veterans, want such consumers to be generic

By generic, I suppose you mean they don't do any specific processing on the data they consume, but just store them in a staging table. In other words, they are just dummy.

  1. If the data you consume has to be processed and again put back to Kafka (i.e. to another topic), it would make sense to write the processing logic in the consumer itself.

    Because, if you write the logic in a Stored Procedure, that would write the processed data into another table and you put that table data back to Kafka for further consumption which would take more time. You might also end up using a Kafka connector like Debezium for pumping data from your DB to Kafka.

  2. Stored procedures can be database dependent. If you change the database, then you would also have to change the stored procedure. On the other hand, if the processing logic is done at the application level and you have ORM, you need not change it much.

Upvotes: 1

Related Questions