CODEWITHSUNDEEP
linuxairflowairflow-scheduler
yguw
yguw

Reputation: 844

How to stop/kill airflow scheduler started in daemon mode

I and new to airflow and accidentally started airflow scheduler in daemon mode. Now, I want to kill the scheduler and possibly restart it. I tried doing 

    sudo kill -9 <list of pids>
    pkill <name>

nothing is happening. When I run

    ps aux | grep 'airflow scheduler'

I see these entries:

    user1   2907  6.0  1.0 329788 62996 ?        Sl   17:37   1:26 /users/user1/anaconda2/bin/python /users/user1/anaconda2/bin/airflow scheduler -D
    user1   2909  0.0  0.9 327576 58948 ?        Sl   17:37   0:00 /users/user1/anaconda2/bin/python /users/user1/anaconda2/bin/airflow scheduler -D
    user1   2910  0.0  0.9 327576 58944 ?        Sl   17:37   0:00 /users/user1/anaconda2/bin/python /users/user1/anaconda2/bin/airflow scheduler -D
    user1   2911  0.0  0.9 327576 58944 ?        Sl   17:37   0:00 /users/user1/anaconda2/bin/python /users/user1/anaconda2/bin/airflow scheduler -D

...and so on for 35 lines with different pids.

Any recommendation as to how I can stop/kill airflow scheduler without restarting my machine. I have also checked the pid file for scheduler and tried killing that pid but no effects.

Any help is appreciated. Thanks!

Upvotes: 15

Views: 40827

Answers (5)

Alejandro Kaspar
Alejandro Kaspar

Reputation: 464

Another alternative is:

/usr/bin/rm -f ${AIRFLOW_HOME}/airflow-scheduler.pid
/usr/bin/pkill -f "airflow-scheduler"

Please note better to remove older daemon .pid file; to avoid any issue while restarting on daemon mode.

You can also see: https://github.com/apache/airflow/issues/77

/bin/sh -c 'PATH=/bin:/sbin:/usr/bin:/usr/sbin  mPPID=`cat ${AIRFLOW_HOME}/airflow-scheduler.pid`;ps -o pid= --ppid $mPPID | xargs kill -15 && kill -15 $mPPID && rm -f ${AIRFLOW_HOME}/airflow-scheduler.pid'"

But this command relies on parent PID (.pid file) so if parent process was removed and child processes are still running it won't work.

So in my opinion accepted answer is the best. Or if you have installed use pkill

If using any monitoring service i.e monit like in the github link. Valid stop command would be:

gPPID=`ps -ef | /usr/bin/grep 'airflow scheduler' | /usr/bin/grep -v grep | awk '{print $2}' `; echo $gPPID; | xargs kill -15 && rm -f ${AIRFLOW_HOME}/airflow-scheduler.pid

Upvotes: 1

spennybuns
spennybuns

Reputation: 247

Go to the airflow directory where the pid file is and use: cat airflow-scheduler.pid | xargs kill

Upvotes: 6

Surya Venkatapathi
Surya Venkatapathi

Reputation: 336

To kill airflow webserver and scheduler, you can use below command

If you have supervisord configured for airflow, then stop it first 

supervisorctl stop all

kill -9 `ps aux | grep airflow | awk '{print $2}'`

Upvotes: 1

Will W
Will W

Reputation: 43

cd ~/airflow
cat airflow-scheduler.pid | xargs kill

Upvotes: 0

venergiac
venergiac

Reputation: 7707

Unfortuntely

kill $(ps -ef | grep "airflow scheduler" | awk '{print $2}')

I was not able to find a clean solution.

Also looking into the code

https://github.com/apache/incubator-airflow/blob/master/airflow/bin/cli.py

Upvotes: 15

Related Questions