CODEWITHSUNDEEP
node.jsgoogle-cloud-platformgoogle-cloud-pubsub
Rusty Gold
Rusty Gold

Reputation: 69

GCP PubSub ordering does not order messages

I’m trying to use the GCP pubsub ordering feature with node js. I’m sending a couple of messages with the same ordering key and after the first message gets acknowledged I’m receiving all other messages in the bunch but not in the order. You can see that behavior on the screen. Take a look at the timestamps. Why does this happen? As far as I understand I should only get another message after acknowledging the previous one. Results

Code for publishing message:

const main = async () => {
  const pubSubClient = new PubSub()

  const keys = ['a', 'a', 'a']
  for (let i = 0; i < keys.length; ++i) {
    console.log(chalk.green(`Sending message ${i + 1}, ordering key - ${keys[i]}`))
    await pubSubClient
      .topic('test-topic', { messageOrdering: true })
      .publishMessage({
        orderingKey: keys[i],
        data: Buffer.from(`Msg ${i + 1}`)
      })
  }
}

Code for pulling messages:

const main = async () => {
  const pubSubClient = new PubSub()

  const handleMsg = async (message: Message) => {
    const dataStr = Buffer.from((message.data as unknown) as string, 'base64').toString('utf-8')

    console.log(chalk.green(`Got message ${dataStr}, orderingKey - ${message.orderingKey}, ${new Date().toISOString()}`))
    // some processing happens here
    await sleep(10000)

    console.log(chalk.yellow(`Message ack ${dataStr}, orderingKey - ${message.orderingKey}, ${new Date().toISOString()}`))
    message.ack()
  }

  const subscription = pubSubClient.subscription('test-subscription')

  subscription.on('message', handleMsg)

  console.log('started listening...')
}

Upvotes: 0

Views: 866

Answers (1)

guillaume blaquiere
guillaume blaquiere

Reputation: 75940

The process works well. There is no issue or problem.

You program is a subscriber connected to a PubSub subscription. A subscription can have multiple subscriber. So, PubSub send a first message to one subscriber (the only one in your case, but PubSub isn't aware of that).

The first message is well acknowledge on that "subscriber", PubSub choose to deliver the others. To optimize the network, PubSub deliver the message in bulk to the client (that's why you have the same timestamp). However, the messages are processed in order in your program; the PubSub client library ensure that order.

At the end of the day, you receive all the messages and process them in order. Job Done.

Upvotes: 1

Related Questions