CODEWITHSUNDEEP
elasticsearchapache-kafkaapache-kafka-connect
Luiza Kharatyan
Luiza Kharatyan

Reputation: 9

Kafka Sink Connector: Is there any way to apply value transformations only for messages that meet the condition?

I'm trying to use Elasticsearch sink connector to transfer all messages to ES index.
There is a drop transformation for Kafka connectors, that tells connector to delete rows from ES if the body is null.

What if for delete action we send a message with not-null body? Is there any way to apply transformation with some condition/predicate, at the same time continue processing create/update messages without transformation? Like to apply drop value transformation only on rows that have deleted flag true in their bodies.

Upvotes: 0

Views: 531

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191894

The transforms don't "delete from ES", they only modify the Kafka record.

If you want to act only on specific records, that's what the Filter w/ Predicate transform is for, which you'd need to chain before a drop transformation since I don't think it's possible to invoke a delete ES event with a non-null record value

at the same time continue processing create/update messages without transformation?

You'd need to run another connector that reverses the predicate condition of the other

Upvotes: 0

Related Questions