CODEWITHSUNDEEP
springapache-kafkakafka-consumer-apicircuit-breaker
Joydeep
Joydeep

Reputation: 405

Circuit breaker with kafka consumer

Is there a way to implement a circuit breaker pattern with Spring Kafka based consumer . I am wondering while implementing my Spring kafka consumer is it possible to stop consuming records if there is a failure to process the data based on some external system and which throws a network error. However if the network issue is resolved the consumer should again process normally.

Upvotes: 6

Views: 13677

Answers (3)

KSD
KSD

Reputation: 99

You can refer this solution if you want to stop consuming messages when downstream service or DB is down.

Example

  • Consumer is calling service A
  • Service A is calling External HTTP service B
  • You want to setup circuit breaker when external Service B is down

In this case you can setup circuit breaker on Service A. Whenever External Service B is down then this circuit will open. Then upon state transition of this circuit breaker, call your listener/bindings(if you are using spring-cloud-stream) to stop/pause consumer. So your messages will stay on queue/topic until the circuit breaker is closed again & you dont have to deadletter messages or put them onto error queue/topic.

You can refer below link for detailed solution which uses Resilience4j for circuit breaker implementation & spring-cloud-stream for consumer.

https://dublincoders.com/circuit-breaker-kafka/

Upvotes: 9

Vassilis
Vassilis

Reputation: 1054

Given:

  • Service A produces to topic T event e1
  • Service B consumes from topic T event e1 and calls remote REST service r1

Then:

  • As service A and B interact via topic T, no need of back-pressure is needed.
  • You need to close the circuit between service B and remote REST service r1. To do so, you can just use the spring boot @HystrixCommand and throw an exception in the "fallbackMethod" method. That will not commit offsets and you will re-consume event e1 as long as the circuit is open.

This is how I have done it :-) If you have found a better way please share!

Upvotes: 1

Michael Qin
Michael Qin

Reputation: 665

Retries with Dead Letter Queue (DLQ) is a good pattern to handle consumer failure, and Circuit Breaker pattern is good to handle producer issues.

Upvotes: 3

Related Questions