CODEWITHSUNDEEP
google-cloud-platformgcloudgoogle-cloud-pubsub
sagie
sagie

Reputation: 3026

Cloud pubsub slow poll rate

I have a pubsub topic, with one subscription, and two different subscribers are pulling from it. Using stackdriver, I can see that the subscription has ~1000 messages. Each subscriber runs the following poll loop:

client = pubsub.Client()
topic = client.topic(topic_name)
subscription = pubsub.Subscription(subscription_name)

while True:
    messages = subscription.pull(return_immediately=True, max_messages=100, client=client)
    print len(messages)
    # put messages in local queue for later processing. Those processes will ack the subsription

My issue is a slow poll rate - even though I have plenty of messages waiting to be polled, I'm getting only several messages each time. Also, lots of responses are back without any messages. According to stackdriver, my messages pulled rate is ~1.5 messages/sec.

I tried to use return_immediately=False, and it improved it a bit - the pull rate increased to ~2.5 messages/sec, but still - not the rate I would expect to have.

Any ideas how to increase pull rate? Any pubsub poll best practices?

Upvotes: 2

Views: 2522

Answers (1)

Kamal Aboul-Hosn
Kamal Aboul-Hosn

Reputation: 17216

In order to increase your pull rate, you need to have more than one outstanding pull request at a time. How many depends on how fast and from how many places you publish. You'll need at least a few outstanding at all times. As soon as one of them returns, create another pull request. That way, whenever Cloud Pub/Sub is ready to deliver messages to your subscriber, you have requests waiting to receive messages.

Upvotes: 2

Related Questions