Reputation: 81
I'm using KafkaJS ( https://kafka.js.org ) to connect to a Kafka cluster. My consumers are responsible for processing tasks which can fail at some time and succeed after waiting for a duration (for example, 1 hour). Before this duration, any attempt to process any task will result to another failure.
I tried to use Pause & Resume method as described at https://kafka.js.org/docs/consuming#a-name-pause-resume-a-pause-resume. However, the consumer restart immediately and keep consuming the failed task again.
How can I pause consuming messages without committing a failed one and restarting the consumer immediately?
Upvotes: 4
Views: 1915
Reputation: 318
An alternative approach could be to save the last fetched offset at some cache or DB and then call the seek method on consumer to start fetching from the last offset.
Here is one implementation.
Upvotes: 0