CODEWITHSUNDEEP
google-cloud-pubsub
Tameem
Tameem

Reputation: 418

PubSub: horizontally scaling subscriber forwarders

From the documentation at https://cloud.google.com/pubsub/architecture#basic_architecture, We understand how pubsub can relay messages between publisher forwarders and subscriber forwarders and the image at the location https://cloud.google.com/pubsub/images/wp_msg_lifecycle.svg visualises this information.

From all the information provided, I could understand that, at any given point in time, the unacknowledged messages in multiple publish message sources can be consumed by a subscriber forwarder and total number of publish message sources depend on the publisher forwarders.

In a scenario where the data is supposed to be consumed in real time, how does pubsub scale the forwarders to achieve no backlog. Because, each publisher forwarder at any given point in time can connect to a single subscriber forwarder and as the scaling of publisher forwarder and subscriber forwarder is independent/decoupled this may increase backlog in the system as subscriber forwarder is supposed to relay multiple messages to the subscribers.

Upvotes: 0

Views: 880

Answers (1)

Kamal Aboul-Hosn
Kamal Aboul-Hosn

Reputation: 17161

There is no limitation that a publisher forwarder can only connect to a single subscriber forwarder, even though that is the example shown in the diagram. The limiting factor in end-to-end latency in Cloud Pub/Sub is very rarely ever the service itself; it is the subscriber client behavior. In particular, it is important to:

  1. Have enough subscriber capacity to handle with the throughput.
  2. Ack messages quickly, on the order of milliseconds or seconds.
  3. Avoid nacking messages, particularly for push subscriptions, where nacks cause backoff in the delivery rate of messages.

Upvotes: 2

Related Questions