CODEWITHSUNDEEP
javascriptnode.jskafka-consumer-apikafkajs
John Glabb
John Glabb

Reputation: 1621

how to tell kafka client to consume messages in order they arrived?

I use kafka client 2.4.4 to consume messages

const { Kafka, logLevel } = require("kafkajs")
const kafka = new Kafka(config)
const consumer = kafka.consumer({ groupId: "Default" })

await consumer.connect()
await consumer.subscribe({ topic, fromBeginning: true })

await consumer.run({
    autoCommitThreshold: 10,
    eachMessage: async ({ topic, partition, message, heartbeat, pause }) => {
       await processMessage(message)
    .....

but looks like consumer picks messages in random order not the order they arrived. Is there any client/consumer option to tell client read it in proper order?

Looks like I can use following method:

eachBatch: async ({ batch

then extract each message and sort them by timestamp but it's not really right approach

Upvotes: 0

Views: 77

Answers (1)

ashish
ashish

Reputation: 318

While producing message to kafka if you specify a key then all the messages will go to same partition of kafka. This way when consuming all the messages should come in order.

Producing with key based partitioning

Upvotes: 0

Related Questions