CODEWITHSUNDEEP
apache-kafkacanary-deployment
Michael Böckling
Michael Böckling

Reputation: 7872

How to run a Kafka Canary Consumer

We have a Kafka queue with two consumers, both read from the same partition (fan-out scenario). One of those consumers should be the canary and process 1% of the messages, while the other processes the 99% remaining ones.

The idea is to make the decision based on a property of the message, eg the message ID or timestamp (e.g. mod 100), and accept or drop based on that, just with a reversed logic for canary and non-canary.

Now we are facing the issue of how to do so robustly, e.g. reconfigure percentages while running and avoid loosing messages or processing them twice. It appears this escalates to a distributed consensus problem to keep the decision logic in sync, which we would very much like to avoid, even though we could just use ZooKeeper for that.

Is this a viable strategy, or are there better ways to do this? Possibly one that avoids consensus?

Update: Unfortunately the Kafka Cluster is not under our control, and we cannot make any changes.

Update 2 Latency of messages is not a huge issues, a few hundred 100ms added are okay and won't be noticed.

Upvotes: 4

Views: 3806

Answers (1)

radai
radai

Reputation: 24192

I dont see any way to change the "sampling strategy" across 2 machines without "ignoring" or double-processing records. Since different Kafka consumers could be in different positions in the partition, and could also get the new config at different times, you'd inevitably run into one of 2 scenarios:

  1. Double processing of the same record by both machines
  2. "Skipping" a record because neither machine thinks it should "own" it when it sees it.

I'd suggest a small change to your architecture instead:

  • Have the 99% machine (the non-canary) pick up all records, then decide for every record if it wants to handle it, or if it belongs to the canary
  • If it belongs to the canary, send the record to a 2nd topic (from the 99% machine)
  • Canary machine only listens on the 2nd topic, and processes every arriving record

And now you have a pipeline setup where decisions are only ever made in one point and no records are missed or double processed.

The obvious downside is somewhat higher latency on the canary machine. If you absolutely cannot tolerate the latency push the decision of which topic to produce to upstream to producers? (I don't know how feasible that is to you)

Variant in case a 2nd topic isnt allowed

If (as youve stated above) you cant have a 2nd topic, you could still make the decision only on the 99% machine, then for records that need to go to the canary, re-produce them into the origin partition with some sort of "marker" (either in the payload or as a kafka header, up to you). The 99% machine will ignore any incoming records with a marker, and the canary machine will only process records with a marker.

Again, the major downside is added latency.

Upvotes: 4

Related Questions