CODEWITHSUNDEEP
airflowgoogle-cloud-composer
Ching-Chien Chen
Ching-Chien Chen

Reputation: 73

Composer auto scaling?

Given that GCP Cloud Composer is running with GKE/GCE, is it auto scaling?

Now I have 3 nodes in the cluster to support say, 100 DAGs.

Later, if I have about 300 DAGs, will it scale up itself (w/ celery workers)?

Upvotes: 7

Views: 2236

Answers (4)

blackbishop
blackbishop

Reputation: 32700

Cloud Composer brings native support for Environment Scaling since major version 2:

gcloud beta composer composer environments update <ENVIRONMENT_NAME> \
--location <LOCATION> \
--min-workers <WORKERS_MIN> \
--max-workers <WORKERS_MAX>

See Scale Environments

Upvotes: 1

Joshua H
Joshua H

Reputation: 774

I just published an article showing how to enable autoscale in its underlying kubernetes https://link.medium.com/AMUlwUIkD0.

Basically:

  1. Enable autoscale on the node level

  2. Apply HorizontalPodAutoscaler to airflow-worker deployment

  3. Increase some airflow config parameter to remove the bottleneck

This has been tested in composer-1.7.2 and composer-1.7.5 but might be applicable to other version as well

Do check it out

Upvotes: 5

Feng Lu
Feng Lu

Reputation: 691

We currently don't support auto-scaling but it's on our roadmap. You could however manually scale up/down the GKE cluster by updating the nodeCount value.

Upvotes: 5

ryw
ryw

Reputation: 9655

No autoscaling at this point, but I expect it's in the roadmap.

Astronomer is working on an autoscaling Airflow service using Kubernetes, it'll probably launch around the time Google Composer launches autoscaling... so good times are coming :)

Upvotes: 1

Related Questions