CODEWITHSUNDEEP
google-cloud-platformterraformairflowgoogle-cloud-composer
duyguevrim
duyguevrim

Reputation: 135

Google Cloud Composer | Airflow - Specifying or closing maintenance windows

I'm using Airflow with Cloud Composer. I have a Dag scheduled to work every hour. However, I read something like this in the composer documentation: "Maintenance operations might impact the execution of your DAGs and Airflow tasks"

But the cron job I created should run for every hour. I don't want any downtime due to maintenance windows. I'm worried about any problems with the selected maintenance windows. Can you give me information about this? Do I have an option to close the maintenance window?

Thanks in advance.

Upvotes: 1

Views: 656

Answers (1)

Ricco D
Ricco D

Reputation: 7287

As you have mentioned, running DAGs on the maintenance window might cause scheduling or execution issues. This was also mentioned in Cloud Composer Troubleshooting Scheduler issues.

You can define specific maintenance windows for your environment. During these time periods, maintenance events for Cloud SQL and GKE take place.

Avoid scheduling DAG runs during maintenance windows because this might cause scheduling or execution issues.

Also removing the maintenance window won't be considered as a feature moving forward as per Cloud Composer Issue Tracker.

Information from the Engineering team is that this feature is not in the road map. Hence, We're not going to have this feature in the future to remove the maintenance window once it's applied to the composer environment.

Reason: If Environment don't have any maintenance windows then maintenance operations are happening at random times and having maintenance window allows to have maintenance operations in predictable slots.

Unfortunately your only option is to deal with the scheduling or execution issues if ever you encounter one. Since you have mentioned that your DAG runs every hour.

Upvotes: 1

Related Questions