CODEWITHSUNDEEP
google-cloud-pubsub
Anil Muppalla
Anil Muppalla

Reputation: 414

Pubsub pull at a steady rate?

Is there a way to enforce a steady poll rate using the google-cloud-pubsub client?. I want to avoid scenarios where if there is spike in the publish rate, the pull request rate also tend to increase.

The client provides FlowControl settings, by setting the maxOutstanding messages. From my understanding, it sets the max batch size during a pull operation.

I want to understand how to create a constant pull rate, say 1000 RPS.

Upvotes: 1

Views: 1087

Answers (1)

rilla
rilla

Reputation: 793

Message Flow Control can be used to set the maximum number of messages being processed at a given time (i.e., setting max_messages in the case of the python client), which indirectly sets the maximum rate at which messages are received. While it doesn’t allow you to directly set the exact number of messages received per second (that would depend on the time it takes to process a message and the number of messages being processed), it should avoid scenarios where you get a spike in publish rate.

If you really need to set a rate in messages received per second, AFAIK it’s not made available directly on the client libraries, so you’d have to implement it yourself using an asynchronous pull and using some timers to acknowledge the messages at your desired rate.

Upvotes: 2

Related Questions