CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-pubsubpublish-subscribemessaging
Aakash
Aakash

Reputation: 15

Ensuring Sequential Delivery with Multiple Ordering Keys in Pub/Sub

I'm working on a Pub/Sub system where I need to ensure that messages are received in the exact sequence they are published, even when using multiple ordering keys. My system publishes a large number of messages, potentially spanning various topics and ordering keys, but it's crucial for the application logic that these messages are processed in the order they were sent.

I have implemented a Pub/Sub model using Google Cloud Pub/Sub, employing ordering keys to ensure that messages with the same key are processed in order. Here's a simplified version of my publishing logic:

Publisher publisher = Publisher.newBuilder(topicName).build();
String[] orderingKeys = {"OrderKey1", "OrderKey2", "OrderKey3"}; // Multiple ordering keys
for (String orderingKey : orderingKeys) {
    for (int i = 1; i <= 500; i++) { // Publish 500 messages for each ordering key
        String messageStr = "Message " + i + " for " + orderingKey;
        ByteString data = ByteString.copyFromUtf8(messageStr);
        PubsubMessage pubsubMessage = PubsubMessage.newBuilder()
                .setData(data)
                .setOrderingKey(orderingKey)
                .build();
        ApiFuture<String> future = publisher.publish(pubsubMessage);
        System.out.println("Published message ID: " + future.get());
    }
}

And my subscriber simply acknowledges the messages as they come in, printing out their content.

Despite the ordering keys ensuring that messages with the same key are ordered, I am facing an issue where messages from different ordering keys are not received in the sequence they were published. For example, I might receive messages in the following sequence:

Received message: Message 1 for OrderKey1 Received message: Message 2 for OrderKey1 Received message: Message 1 for OrderKey2 Received message: Message 3 for OrderKey1 Received message: Message 2 for OrderKey2

This is problematic for my use case as I need the messages to be processed exactly in the sequence they were published, regardless of the ordering key.

I am looking for a solution or design pattern that allows me to maintain the global ordering of messages across multiple ordering keys. Ideally, messages should be received and processed in the exact sequence they were published by the publisher, irrespective of the ordering key used.

Is there a way to achieve this in Google Cloud Pub/Sub, or should I consider an alternative approach or solution to meet this requirement?

Upvotes: 1

Views: 302

Answers (1)

Kamal Aboul-Hosn
Kamal Aboul-Hosn

Reputation: 17261

If you are trying to maintain global ordering across all messages, then ordering keys is not going to be a good choice because, as you observe, order is only guaranteed within a key, not across keys. If you want total ordering with Pub/Sub, you will need to use something like Dataflow windowing to emit messages grouped in ordered time windows by publish time. Note that if you use Dataflow, you should not use ordering keys as it interferes with Dataflow's windowing.

Upvotes: 0

Related Questions