CODEWITHSUNDEEP
google-compute-enginegoogle-cloud-pubsubgce-instance-group
TimK
TimK

Reputation: 829

Autoscaling GCE Instance groups based on Cloud pub/sub queue

Can GCE Instance groups be scaled up/down bases on Google Cloud PubSub queue counts or other asynchronous task queues such as PSQ?

Upvotes: 3

Views: 2387

Answers (3)

Kevin Moore
Kevin Moore

Reputation: 6161

Yes!

The feature is now in alpha: https://cloud.google.com/compute/docs/autoscaler/scaling-queue-based

Upvotes: 5

sagie
sagie

Reputation: 3026

You can't use pubsub metrics (pubsub.googleapis.com/subscription/num_outstanding_messages or pubsub.googleapis.com/subscription/num_undelivered_messages) for that purpose. According to the docs:

A valid utilization metric for scaling meets the following criteria:

  1. The standard metric has a label for resource_id, and value of the label for each stream is ID of an instance.

  2. The standard metric describes how busy an instance is, and the metric value increases or decreases proportionally to the number virtual machine instances in the group.

pubsub metrics don't meet that criteria.

However, there are two ways you can use pubsub based autoscaling:

  1. Write your own custom metric - you can use the gcloud monitoring api to get your pubsub timeseries data. Than use it to calculate your own custom monitoring metric - for example - last time series value divided by your average/desired latency. You can use this method with every async queue solution that you are using.

  2. Still in alpha, there is a gcloud api for subscriber based autoscale: https://cloud.google.com/compute/docs/autoscaler/scaling-queue-based. This solution applies for google cloud pubsub only, and you can't use it with other async queue solutions.

Upvotes: 1

Mete Atamel
Mete Atamel

Reputation: 831

I haven't tried this myself but looking at the documentation, it looks possible to set up autoscaling against Pub/Sub message queue counts.

This page [0] explains how to setup autoscaler to scale based on a standard metric provided by the Cloud Monitoring service.

This page [1] explains what metrics you can use for autoscaler. These two looks useful:

pubsub.googleapis.com/subscription/num_outstanding_messages pubsub.googleapis.com/subscription/num_undelivered_messages

[0] https://cloud.google.com/compute/docs/autoscaler/scaling-cloud-monitoring-metrics [1] https://cloud.google.com/monitoring/api/metrics

Upvotes: 2

Related Questions