CODEWITHSUNDEEP
airflowairflow-scheduler
AnupaM
AnupaM

Reputation: 197

how to kill airflow scheduler and webserver?

I am new to airflow, tried to run a dag by starting airflow webserver and scheduler. After I closed the scheduler and airflow webserver, the airflow processes are still running.

ps aux | grep airflow shows 2 airflow webserver running, and scheduler running for all dags.

I tried running kill $(ps aux | grep airflow | awk '{print $2}') but it did not help.

I don't have sudo permissions and webserver UI access. enter image description here

Upvotes: 8

Views: 33529

Answers (2)

SergiyKolesnikov
SergiyKolesnikov

Reputation: 7815

If you run Airflow locally and start it with the two commands airflow scheduler and airflow webserver, then those processes will run in the foreground. So, simply hitting Ctrl-C for each of them should terminate them and all their child processes.

If you don't have those two processes running in the foreground, there is another way. Airflow creates files with process IDs of the scheduler and gunicorn server in its home directory (by default ~/airflow/).

Running

kill $(cat ~/airflow/airflow-scheduler.pid)

should terminate the scheduler.

Unfortunately, airflow-webserver.pid contains the PID of the gunicorn server and not the initial Airflow command that started it (which is the parent of the gunicorn process). So, we will first have to find the parent PID of the gunicorn process and then kill the parent process.

Running

kill $(ps -o ppid= -p $(cat ~/airflow/airflow-webserver.pid))

should terminate the webserver.

If simply running kill (i.e., sending SIGTERM) for these processes does not work you can always try sending SIGKILL: kill -9 <pid>. This should definitely kill them.

NOTE: Starting with Airflow 2.2.0, there is the standalone CLI command that runs all Airflow components, which are needed for local development. The command can be terminated with simple Ctrl-C and it automatically shuts down all running components.

Upvotes: 11

ksolemon
ksolemon

Reputation: 21

if you are running Airflow Webserver in the background with (airflow webserver -d -p 8080) command you will be able to use existing terminal/Ubuntu window for further commands. In this case use (airflow webserver) command. You will receive a message that Airflow Server is already running with PID xxx. Take that PID and use (kill xxx) command while replace xxx with PID (also called Process ID) and you will be done.

If you are running Airflow Webserver with (airflow webserver) command, then you are running Airflow Webserver in the foreground, and may not be able to use existing terminal/Ubuntu window. In this case try using (Ctrl + c) command inside the terminal that is running Airflow Server. You won't see your name prompt, but still a blinking cursor, click by the blinking cursor and press (Ctrl + c). If this works, then you are done closing Airflow Webserver. If not then open another terminal/Ubuntu window and use (airflow webserver) command. You will receive a message that Airflow Server is already running with PID xxx. Thake that PID and use (kill xxx) command while replace xxx with PID (also called Process ID) and you will be done.

Upvotes: 0

Related Questions