CODEWITHSUNDEEP
airflowgoogle-cloud-composer
Diego Stranger
Diego Stranger

Reputation: 43

Is it possible to turn off the VM´s hosting the google-cloud-composer at certain hours?

In order to reduce the billing associated of running the google-cloud-composer, I am wondering about the possibility to turn off the VM instances that run the Virtual Environment at certain hours. For example: Most of our DAG´s run either in the morning or the afternoon, so we would like to turn off the VM´s during the night, or even during mid-day if is it possible. I know we can disable the environments manually from the Google cloud console, but it would be great to find a way to do this automatically

Thanks!

Upvotes: 3

Views: 3491

Answers (2)

HulaHoof
HulaHoof

Reputation: 399

We have set up a tiny k8s cluster in parallel and use a CronJob deployment to manage scaling the pool nodes down to 0, then a second cronjob to bring it back up.

Deployment is something like this (changing node count for up and down). You could change the action to however it is you are treating the VMs (do you pause the instances via Compute in the console?)

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: disable-composer
spec:
  schedule: "0 10 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: enable-composer
            image: google/cloud-sdk:latest
            volumeMounts:
            - name: google-app-credentials-volume
              mountPath: /etc/gcp
              readOnly: true
            env:
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: /etc/gcp/credentials.json
            args:
            - /bin/bash
            - -c 
            - gcloud auth activate-service-account [email protected] --key-file=$GOOGLE_APPLICATION_CREDENTIALS; COMPOSER_ENV=composer-environment-name; COMPOSER_LOCATION=us-central1; COMPOSER_CLUSTER=`gcloud composer environments describe $COMPOSER_ENV --format="csv[no-heading](config.gkeCluster)" --location $COMPOSER_LOCATION | cut -d '/' -f 6`; COMPOSER_ZONE=`gcloud composer environments describe $COMPOSER_ENV --format="csv[no-heading](config.nodeConfig.location)" --location $COMPOSER_LOCATION | cut -d '/' -f 4`; gcloud container clusters resize $COMPOSER_CLUSTER --zone $COMPOSER_ZONE --size=0 --quiet;
          restartPolicy: OnFailure
          volumes:
          - name: google-app-credentials-volume
            secret:
              secretName: google-app-credentials
              items:
              - key: credentials.json
                path: credentials.json

Where google-app-credentials is a kubernetes secret containing our service account keyfile.

Upvotes: 2

Patrick W
Patrick W

Reputation: 4909

Unfortunately there is no way to have this configured programatically using the Google Cloud Platform. Your best option for this would be to have a script run as a cronjob from another host that will turn up or turn down the Composer environment when it is not in use.

Upvotes: 3

Related Questions