Can I set Kafka consumer as a cron job?

Question

I have a simple process that uses Kafka, with 1 producer and 1 consumer. The producer will continuously run, while the consumer should only run on a specific time range within a day. Is this possible? And can I schedule the job (the consumer process) using Control-M? It should stop until all queued messages are completely processed. Then consumer will run again as scheduled.

Answer 1

Yes, you can do this. Camus/Gobblin/Spark Structured Batch are all examples of Kafka processes that already do this

If you must write your own script, then you can use kafka-console-consumer with --max-messages flag to ensure the consumer doesn't remain open forever

Keep in mind that if the producer is constantly running, and the consumer is not, then it'll get further and further behind, so your scheduled consumer may never reach the latest offsets... Therefore, this is not recommended for high volume topics