CODEWITHSUNDEEP
airflowairflow-schedulergoogle-cloud-composer
cryanbhu
cryanbhu

Reputation: 5254

Do I need to be concerned with Airflow scheduler details for Google Cloud Composer?

In Airflow scheduler, there are things like heartbeat and max_threads.
See How to reduce airflow dag scheduling latency in production?.

If I am using Google Cloud Composer, do I have to worry/set these values?
If not, what are the values that Google Cloud Composer uses?

Upvotes: 1

Views: 530

Answers (2)

Alessandro
Alessandro

Reputation: 655

You can see the airflow config in the composer instance bucket gs://composer_instance_bucket/airflow.cfg. You can tune this configuration as you wish, keeping in mind that cloud composer has some configurations blocked. Also, if you go in the Airflow UI -> Admin -> Configuration you can see the full configuration.

Upvotes: 1

ryw
ryw

Reputation: 9655

If you'd like more control/visibility of these variables, consider hosted Airflow @ Astronomer https://www.astronomer.io/cloud/ as it runs vanilla Airflow.

Upvotes: 0

Related Questions