CODEWITHSUNDEEP
jsonsql-serverapache-kafkaapache-kafka-connectdebezium
Mujtaba Faizi
Mujtaba Faizi

Reputation: 339

Filter the records by a certain value in kafka connect

I have been trying to find a way to include a filter that discards the records with a particular value, in the debezium source sql connector. Lets say, I have the JSON: {"id":0, "name":"muj"} and I want to discard all the records that have the field "name" value as "muj". Is there a way to do that without having to use confluent platform?

Upvotes: 4

Views: 8072

Answers (2)

kikulikov
kikulikov

Reputation: 2582

You could use the filter function of Kafka Connect transformations (the one from Confluent). The filter function allows to include or exclude records that match the predicate based on record values. Please, see transforms/filter documentation and examples.

Upvotes: 4

OneCricketeer
OneCricketeer

Reputation: 191993

Is there a way to do that without having to use confluent platform

Confluent is not a special version of Kafka. All their tools will work with your own Kafka installation.

You can use ksql or Kafka Streams individually, but IMO, Debezium shouldn't be filtering anything, it's meant to read raw CDC events. Any such filtering should be done downstream in your processing logic

Upvotes: 3

Related Questions