CODEWITHSUNDEEP
node.jspublish-subscribegoogle-cloud-pubsub
Rimon Fedyuk
Rimon Fedyuk

Reputation: 491

GCP PubSub - Batching with orderingKey (Nodejs)

There is an unstable behavior while using google PubSub batch with orderingKey. If I use batch PubSub without orderingKey, the code work as expected:

        console.log( `Start: [${new Date().toISOString()}]` );
    
        const array = [ ...Array( 5 ).keys() ];
        const topic = pubSubClient.topic( 'topic_test', {
            batching: {
                maxMessages: array.length,
                maxMilliseconds: 10 * 1000,
            },
        } );

        await Promise.all(
            array.map( async ( item ) => {
                const messageId = await topic.publish( Buffer.from( JSON.stringify( item ) ) );
                console.log( `[${new Date().toISOString()}] Message ${messageId} published. id: ${item}` );
            } )
        );

I get an immediate result without waiting:

Start: [2021-06-22T07:47:01.187Z]
[2021-06-22T07:47:01.187Z] Message 2578553779341346 published. id: 0
[2021-06-22T07:47:01.188Z] Message 2578553779341347 published. id: 1
[2021-06-22T07:47:01.188Z] Message 2578553779341348 published. id: 2
[2021-06-22T07:47:01.188Z] Message 2578553779341349 published. id: 3
[2021-06-22T07:47:01.188Z] Message 2578553779341350 published. id: 4

But if I add an orderingKey with batch together:

        console.log( `Start: [${new Date().toISOString()}]` );
        
        const array = [ ...Array( 5 ).keys() ];
        const topic = pubSubClient.topic( 'topic_test', {
            enableMessageOrdering: true,
            batching: {
                maxMessages: array.length,
                maxMilliseconds: 10 * 1000,
            },
        } );

        await Promise.all(
            array.map( async ( item ) => {
                const messageId = await topic.publishMessage( {
                    data: Buffer.from( JSON.stringify( item ) ),
                    orderingKey: item,
                } );
                console.log( `[${new Date().toISOString()}] Message ${messageId} published. id: ${item}` );
            } )
        );

The batch sent only after 10 seconds (by maxMilliseconds), even though the maxMessages condition has been fulfilled before:

Start: [2021-06-22T07:54:31.287Z]
[2021-06-22T07:54:41.995Z] Message 2578621698696149 published. id: 1
[2021-06-22T07:54:41.995Z] Message 2578554178769363 published. id: 4
[2021-06-22T07:54:41.996Z] Message 2578622069514679 published. id: 3
[2021-06-22T07:54:41.996Z] Message 2578621502622110 published. id: 0
[2021-06-22T07:54:42.058Z] Message 2578621667693212 published. id: 2

How can I fix it?

Upvotes: 0

Views: 475

Answers (1)

Kamal Aboul-Hosn
Kamal Aboul-Hosn

Reputation: 17251

When using ordering keys, batching is done per ordering key. Therefore, if each message has a unique ordering key as is likely in your case, you will always end up waiting maxMilliseconds. Typically, there should be some overlap in ordering key, e.g., it should be a user ID or the unique key of a row in a database. Using the entire item as the ordering key is unexpected.

Upvotes: 1

Related Questions