CODEWITHSUNDEEP
airflow
RonanT
RonanT

Reputation: 165

Run Airflow Webserver and Scheduler on different servers

I was wondering if the Airflow's scheduler and webserver Daemons could be launched on different server instances ?

And if it's possible, why not use serverless architecture for the flask web server ?

There is a lot of resources about multi nodes cluster for workers but I found nothing about splitting scheduler and webserver.

Has anyone already done this ? And what may be the difficulties I will be facing ?

Upvotes: 1

Views: 1494

Answers (1)

SergiyKolesnikov
SergiyKolesnikov

Reputation: 7815

I would say the minimum requirement would be that both instance should have

  • Read(-write) access to the same AIRFLOW_HOME directory (for accessing DAG scripts and the shared config file)
  • Access to the same database backend (for accessing shared metadata)
  • Exactly the same Airflow version (to prevent any potential incompatibilities)

Then just try it out and report back (I am really curious ;) ).

Upvotes: 1

Related Questions