CODEWITHSUNDEEP
airflowgoogle-cloud-composer
MohamedLEGH
MohamedLEGH

Reputation: 311

Start and stop Airflow?

I'm testing airflow google composer but right now I don't use it all the time and it cost money to maintains the 3 VM. Is there a way to stop/start google composer/airflow ?

Is it possible to do it programmatically ? (I want to start it every morning at 9am and stop it every night at 9pm and the week-end)

Right now to stop most of the costs I change the number of VM in Kubernetes Engine (I switch from 3 to 0 to stop it and then to 0 to 3 to start it) but maybe there is a better way.

Upvotes: 3

Views: 2056

Answers (4)

DKM
DKM

Reputation: 1801

It's not possible to start and stop the composer instance as it runs Kubernetes behind the scene and the only way is to delete the instance as soon as you finish your jobs.

Upvotes: 0

André Rocha Silva
André Rocha Silva

Reputation: 1

I am facing the same problem. From my studies, the best solution is to have another VM running a CRON job that turns off and on the environment.

As it takes some time to turn the environment on, if you have to run a task hourly I think the best option is to keep the environment on.

Upvotes: 0

Blessy
Blessy

Reputation: 520

It is not possible to change the number of GKE nodes in Composer environment to 0. It expects minimum 3 nodes.

Upvotes: 0

Wilson
Wilson

Reputation: 614

There is no officially-supported mechanism for pausing a Composer environment. Even though you are able to reduce your VM costs by reducing the cluster size, you continue to be billed for the App Engine and Cloud SQL costs.

Upvotes: 1

Related Questions