CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-composer
Shawn
Shawn

Reputation: 13

Google Cloud Composer 2 shows Unhealthy

Anybody knows why is this happening? Which api composer calls to calculate health? What is this events?

GCP cloud composer 2 shows unhealthy but everything works. Seems like the service account cannot access an API to calculate health metrics. Composer version composer-2.0.25-airflow-2.2.5

"Error from server (Forbidden): events is forbidden: User "xxxxx.iam.gserviceaccount.com' cannot list resource "events" in API group "" in the namespace airflow-2-2-5-11111111": GKEAutopilot authz: the request was sent before policy enforcement is enabled" timestamp : '2022-10-07T15:48:08.834865950Z" composer

Upvotes: 0

Views: 711

Answers (1)

Prajna Rai T
Prajna Rai T

Reputation: 1810

The environment health metric depends on a Composer-managed DAG named airflow_monitoring which is triggered periodically by the airflow-monitoring pod.

The airflow_monitoring DAG is a per-environment liveness prober/healthcheck that is used to populate the Cloud Composer monitoring metric environment/healthy. It is an indicator for the general overall health of your environment, or more specifically, its ability to schedule DAGs and run tasks. This allows you to use Google Cloud Monitoring features such as metric graphs, or setting alerts when your environment becomes unhealthy.

If this DAG isn't deleted, you can check the airflow-monitoring logs to see if there are any problems related to reading the DAG's run statuses. Consequently, you can also try troubleshooting the error in Cloud Logging using the filter:

resource.type="cloud_composer_environment"
severity=ERROR

You can find more information about the metric on the GCP Metrics List, and can explore the metric in Cloud Monitoring.

Also, you can refer to this documentation for information on Cloud Composer’s environment health metric.

Upvotes: 2

Related Questions