Distribute 1 Pub/Sub Topic Messages across multiple Cloud Functions without repetition

Question

I have 8 Cloud functions that are triggered by a Pub/Sub topic.

Example problem: When 16 messages are published to the Pub/Sub topic, all 8 functions are triggered, and each of the 8 functions receives all 16 messages.

Desired outcome: When 16 messages are published to the Pub/Sub topic, all 8 functions are triggered, each of the 8 functions should receive only 2 messages.

Is there a way to solve this?

I tried: setup 1 pub/sub topic, 1 pub/sub subscription, trigger 8 functions from that 1 subscription, but in yaml documentation and in the cloud console it appears I can only have a function triggered from a topic, not an existing subscription.

Possible but not desired solution: Load balance the (example) 16 messages across 8 different pub/sub topics, then each function gets triggered by 1/8 topics (2 messages per topic).

Answer 1

It's an anti-pattern to have 8 separate Cloud Functions over which you want to load balance messages. If the code is identical for each one, then you should really only have a single Cloud Function and let its autoscaling behavior take care of turning up instances as needed to handle the incoming messages.

If the 8 Functions have different code, then it's counter-intuitive that you want each to only handle a subset of the messages unless you want them routed to the functions based on some property of the messages themselves, in which case you'd want to set up 8 subscriptions that use filters.

If the behavior you seek is truly what you need, then you may need to put a separate Cloud Function in between that receives all messages and distributes it to other Cloud Functions that instead of using Pub/Sub as a trigger, use an HTTP request. Then, you could make the HTTP request from the single function that receives Pub/Sub messages and distribute them as needed across the other 8 functions.