Talend tKafkaInput Consistently Reads Only Partial Messages from Kafka DLQ Despite Correct Configuration

Question

Question: I'm encountering a persistent issue with my Talend jobs involving Kafka and a Dead-Letter Queue (DLQ). I have a real-time job that continuously writes failed messages to a Kafka topic named kafka_dlq. I then have a separate Talend job that runs hourly to process these failed messages from the kafka_dlq.

Problem:

My real-time job successfully writes 1316 messages to the kafka_dlq topic. However, my hourly DLQ processing job, using tKafkaInput, consistently reads only 944 messages, no matter how many times I run it. It seems to be getting stuck at a certain offset.

Key Requirements:

The DLQ processing job must process all messages that were in the kafka_dlq topic at the time the job starts. It should not process messages that arrive in the DLQ while the processing job is running. The kafka_dlq topic has only one partition.

Kafka Configuration (Advanced Settings in tKafkaInput):

auto.offset.reset: earliest enable.auto.commit: false endingOffsets: context.endOffset session.timeout.ms: 30000 heartbeat.interval.ms: 3000 fetch.min.bytes: 4096 fetch.max.wait.ms: 1000 max.poll.interval.ms: 300000 max.poll.records: 1000