CODEWITHSUNDEEP
airflow-scheduler
Rayne
Rayne

Reputation: 14977

Running multiple schedulers in Airflow 2.0 on same machine

I understand that Airflow 2.0 now supports running multiple schedulers concurrently for HA. Can I run multiple schedulers on the same machine (VM)? Do I do it just by running airflow scheduler twice if I want 2 schedulers, without configuring anything else?

Upvotes: 1

Views: 3413

Answers (1)

Greetlist
Greetlist

Reputation: 33

Can I run multiple schedulers on the same machine (VM)?

Yes, you can.The second scheduler can run with like this:

airflow scheduler --skip-serve-logs

Can I run multiple schedulers on the differnt machine (VM)?

Yes, you can.I do create second VM to run multi schedulers.

Check these Dependencies

  1. Set use_row_level_locking config to True(default is True). enter image description here

  2. Check Your backend database's version. enter image description here

  3. Make Sure All scheduler's database config point to same databse.

After check these Dependencies.You can run multi schedulers on different machines.

After I start two schedulers on different VM, I run a task to check whether this task would be executed twice.Fortunately, only one scheduler get this task.

If you run only one webserver, you would loss some task's log when these task is executed on other scheduler machine. At this situation, you would need log collector such as elastic-search.

Upvotes: 2

Related Questions