CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-functionsmessage-queuepublish-subscribegoogle-cloud-pubsub
sheldonzy
sheldonzy

Reputation: 5971

Rate limit GCP Cloud Function triggers from pub/sub topic

I have a Cloud Function that is being triggered from a Pub/Sub topic.

I want to rate limit my Cloud Function, so I set the max instances to 5. In my case, there will be a lot more produced messages than Cloud Functions (and I want to limit the number of running Cloud Functions).

I expected this process to behave like Kafka/queue - the topic messages will be accumulated, and the Cloud Function will slowly consume messages until the topic will be empty.

But it seems that all the messages that did not trigger cloud function (ack), simply sent a UNACK - and left behind. My subscription details:

enter image description here

The ack deadline max value is too low for me (it may take a few hours until the Cloud Function will get to messages due to the rate-limiting).

Anything I can change in the Pub/Sub to fit my needs? Or I'll need to add a queue? (Pub/Sub to send to a Task Queue, and Cloud Function consumes the Task Queue?).

BTW, The pub/sub data is actually GCS events. If this was AWS, I would simply send S3 file-created events to SQS and have Lambdas on the other side of the queue to consume.

Any help would be appreciated.

Upvotes: 8

Views: 2048

Answers (2)

sheldonzy
sheldonzy

Reputation: 5971

The ideal solution is simply to change the retrying policy. When using "Retry after exponential backoff delay", the Pub/Sub will keep retrying even after the maximum exponential delay (600 seconds). This way, you can have a lot of messages in the Pub/Sub, and take care of them slowly with a few Cloud Functions - which fits our need of rate-limiting.

Basically, everything is the same but this configuration changed, and the result is:

enter image description here

Which is exactly what I was looking for :)

Upvotes: 3

MBHA Phoenix
MBHA Phoenix

Reputation: 2217

You cannot compare to kafka because your kafka consumer is pulling messages at its convenience, while Cloud Function(CF) creates a push subscription that is pushing messages to your CF. So some alternatives:

  1. Create a HTTP CF triggered by cloud scheduler that will pull messages from your PULL subscription. Max retention of unack messages are 7 days (hope it's enough)
  2. Use Cloud for which you increase max concurrency (max concurrent request), with proper sizing for CPU and RAM. Of course your can control the max number of cloud run instances (different from max concurrency). And Use PUSH subscription pushing to cloud run. But here Also you will be limited by 10 minutes ack deadline.

Upvotes: 1

Related Questions