CODEWITHSUNDEEP
python-3.xairflow
Mr. President
Mr. President

Reputation: 1549

Restarting the airflow scheduler

I'm trying to get airflow working to better orchestrate an etl process. When I make changes to a dag in my dags folder, I often have to restart the scheduler with

airflow scheduler

before the changes are visible in the UI. I would like to run the scheduler as a daemon process with 

airflow scheduler -D

but we I try to do so, I get a message saying 

[2018-10-17 14:13:54,769] {jobs.py:580} ERROR - 
Cannot use more than 1 thread when using sqlite. Setting max_threads to 1

I think this error pops up because the scheduler is already running as a daemon. However, when I try to find out where the scheduler is being run with

lsof -i

I don't get any results.

Question: Why am I not able to restart the scheduler with airflow scheduler -D. Why does the scheduler restart with airflow webserver? How do I successfully kill the process that is preventing me to run airflow scheduler -D?

Upvotes: 8

Views: 41385

Answers (3)

Ashwin Gupta
Ashwin Gupta

Reputation: 31

If you just restart your webserver, the dag changes gets reflected in UI. No need to restart scheduler for the same. Applicable for 1.8 and 1.10.3. Cant comment for latest 1.10.10.

Upvotes: 0

Magic Draggn
Magic Draggn

Reputation: 45

You need to clear out the airflow-scheduler.pid file at $AIRFLOW_HOME. The stale pid file from the daemon will prevent you to start another scheduler process.

Upvotes: 2

kaxil
kaxil

Reputation: 18844

Run ps aux | grep airflow and check if airflow webserver or airflow scheduler processes are running. If they are kill them and rerun using airflow scheduler -D

Upvotes: 11

Related Questions