CODEWITHSUNDEEP
apache-flink
Dagang Wei
Dagang Wei

Reputation: 26458

How do I scale up/down a running Flink cluster?

I'm running Flink on Kubernetes now. I assume if I update the replicas of TaskManager deployment, Kubernetes scales up/down the number of TM pods for me, but I'm not sure if that is all I need to do. Do I need to do anything else to make the job adapt to the more/less TMs?

Upvotes: 1

Views: 2158

Answers (1)

David Anderson
David Anderson

Reputation: 43419

Apache Flink does not, by default, rescale in response to changes in the number of task managers.

There are various schemes for how Flink rescales in a K8s environment. One, referred to as "active mode", is where Flink knows what resources it wants, and works with K8s to obtain/release resources accordingly. See how to deploy Flink natively on Kubernetes for details.

Another scheme, referred to as "reactive mode", is what you appear to be hoping for: Flink scales itself to use whatever resources have been made available. See Elastic Scaling for details.

You can also deploy Flink on Kubernetes in what is effectively a containerized version of a standalone deployment. In such an environment, rescaling involves these steps:

  1. Stop the job while taking a savepoint
  2. Resume the job from the savepoint, having arranged for the new cluster to be appropriately sized

The Flink Operations Playground has a step-by-step example of how to do rescaling in a docker-compose-based deployment.

Upvotes: 4

Related Questions