Airflow setup for high availability

Question

How to deploy apache airflow (formally known as airbnb's airflow) scheduler in high availability?

I am not asking about the backend DB or RabbitMQ that should obviously be deployed in high availability configuration.

My main focus is the scheduler - is there something special needs to be done?

Answer 1

NOTE: This answer only applies to Airflow >= 2.0.

According to the docs, it appears to be as simple as starting a second airflow-scheduler on another node:

The short version is that users of PostgreSQL 10+ or MySQL 8+ are all ready to go – you can start running as many copies of the scheduler as you like – there is no further set up or config options needed.

See this blog post from Astronomer about the work they did to make the scheduler support running in a HA capacity.

Answer 2

In my scenario, I have 2 schedulers (on 2 separate docker swarms), with the standby cluster scheduler turned off (using docker swarm service scale=0). I needed to make sure the primary scheduler had stopped fully before I started up the standby scheduler. What I found was that having 2 running schedulers (even for a brief time period) resulted in an occasional DAG scheduled to run on both clusters leading to duplicate reports generated from two different cluster zone.

Answer 3

My personal experience was to follow the instructions I found for some best practices; that is to restart the scheduler every 10 runs ( -N 10 ) and use this software when possible:

https://github.com/teamclairvoyant/airflow-scheduler-failover-controller

I also use a DAG which pings a monitoring system to be sure that the scheduler has not gone away.

Answer 4

After a bit digging I found that it is not safe to run multiple schedulers simoultanously, this means that out of the box - airflow schedulers are not safe to use in high availablity environments.

The airflow team are planning to solve this issue by adding a lock mechanism on the DAG data structure, but this is not implemented yet (I checked by running 2 schedulers and saw that they schedule the same dag instances which is not good). This is described here: https://groups.google.com/forum/#!topic/airbnb_airflow/-1wKa3OcwME

I did found a way to workaround this high availalbilty issue by wrapping the schedulers with my own code and use cluster tools for leader election (I personanlly use consul for this purpose). This way only the elected master is running the scheduler and when the master is down the slave replaces him.

Please consider this when u use airflow in high availabilty environments since out of the box, airflow scheduler is currently not suitable for this (unless you solve this issue yourself).

Edit - an alternative approach to the master slave solution is to use a cluster manager/scheduler to make sure that only one airflow scheduler instance is always available. This approach relies on the self healing abilities of the cluster manager u have. For example both mesos and nomad supports this kind of configuration (I presonally chose nomad for its simplicity).