CODEWITHSUNDEEP
ubuntuairflow
Joseph Noirre
Joseph Noirre

Reputation: 387

Airflow Scheduler

Hi I'm currently trying to set up the simplest version of Airflow with SequentialExecutor on ubuntu 16.04 on an EC2 cluster.

I've created a dag to run every 5 minutes, but the dag runs gets stuck in queue unless I ssh into the server, run airflow scheduler and leave that terminal window open.

Is there a way to not have to leave the window open?

Upvotes: 1

Views: 402

Answers (2)

Antoine Augusti
Antoine Augusti

Reputation: 1608

As Daniel said, you will need to daemonize the scheduler so that it keeps running in the background. You can have a look at the Airflow documentation with systemd / upstart.

Upvotes: 3

Daniel Huang
Daniel Huang

Reputation: 6548

You can run the scheduler as a daemon by passing it the --daemon flag. It will run as a background process so it remains active even after you've closed your window. The same flag can be used on the worker and webserver.

-D, --daemon          Daemonize instead of running in the foreground

Alternatively, you can also run airflow as a service with systemd. Airflow has some example config files that you can reference.

Upvotes: 3

Related Questions