delalma
Reputation: 928
Apache airflow run too many processes
I have all the following processes which are running with Airflow. I am trying to understand why there are so many with some duplicated and if they are all needed.
I am running a local executor with postgres and it run with systemd as explain by the following tutorial. https://towardsdatascience.com/an-apache-airflow-mvp-complete-guide-for-a-basic-production-installation-using-localexecutor-beb10e4886b2
I want to better understand the processes to debug it because it seems that the scheduler is not working currently. Any help would be very appreciated.
ubuntu 30932 0.0 0.0 76692 900 ? Ss 00:53 0:00 /lib/systemd/systemd --user
ubuntu 30933 0.0 0.1 259528 2740 ? S 00:53 0:00 (sd-pam)
root 525 0.0 0.3 107992 7280 ? Ss 01:26 0:00 sshd: ubuntu [priv]
ubuntu 619 0.0 0.3 108228 6120 ? S 01:26 0:00 sshd: ubuntu@pts/0
ubuntu 626 0.0 0.2 23380 5388 pts/0 Ss+ 01:26 0:00 -bash
root 19090 0.0 0.0 0 0 ? I 04:04 0:00 [kworker/u4:1]
root 15692 0.0 0.3 107992 7340 ? Ss 04:44 0:00 sshd: ubuntu [priv]
ubuntu 15809 0.0 0.1 107992 3684 ? S 04:44 0:00 sshd: ubuntu@pts/1
ubuntu 15810 0.0 0.2 23380 5356 pts/1 Ss+ 04:44 0:00 -bash
root 20272 0.0 0.0 0 0 ? I 04:50 0:00 [kworker/u4:0]
root 20274 0.0 0.0 0 0 ? I 04:50 0:00 [kworker/0:0]
root 20676 0.0 0.3 107992 7416 ? Ss 04:51 0:00 sshd: ubuntu [priv]
ubuntu 20783 0.0 0.3 108184 6200 ? S 04:51 0:00 sshd: ubuntu@pts/2
ubuntu 20784 0.0 0.2 23380 5376 pts/2 Ss 04:51 0:00 -bash
root 22974 0.0 0.0 0 0 ? I 04:54 0:00 [kworker/1:0]
ubuntu 23001 1.2 4.7 374404 95696 ? Ss 04:54 0:04 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23002 2.2 6.2 501048 126200 ? Ss 04:54 0:07 /usr/bin/python3 /home/ubuntu/.local/bin/airflow webserver
postgres 23030 0.0 0.8 321700 16512 ? Ss 04:54 0:00 postgres: 10/main: airflow airflow ***.*.*.*(53558) idle
postgres 23031 0.1 0.9 322608 19976 ? Ss 04:54 0:00 postgres: 10/main: airflow airflow ***.*.*.*(53560) idle
ubuntu 23038 0.6 4.2 1636692 84520 ? Sl 04:54 0:02 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23047 0.0 0.0 0 0 ? Z 04:54 0:00 [/usr/bin/python] <defunct>
ubuntu 23048 0.0 0.0 0 0 ? Z 04:54 0:00 [/usr/bin/python] <defunct>
ubuntu 23049 0.0 0.0 0 0 ? Z 04:54 0:00 [/usr/bin/python] <defunct>
ubuntu 23052 0.0 3.9 372588 78492 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23053 0.0 3.9 372588 78492 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23056 0.0 3.9 372588 78492 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23058 0.0 3.9 372588 78428 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23059 0.0 3.9 372588 78428 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23063 0.0 3.9 372588 78436 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23064 0.0 3.9 372588 78436 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23069 0.0 3.9 372588 78440 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23072 0.0 3.9 372588 78440 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23074 0.0 3.9 372588 78440 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23077 0.0 3.9 372588 78440 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23080 0.0 3.9 372588 78440 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23082 0.0 3.9 372588 78440 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23086 0.0 3.9 372588 78440 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23089 0.0 3.9 372588 78444 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23091 0.0 3.9 372588 78444 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23093 0.0 3.9 372588 78448 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23097 0.0 3.9 372588 78448 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23099 0.0 3.9 372588 78448 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23103 0.0 3.9 372588 78448 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23104 0.0 3.9 372588 78448 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23108 0.0 3.9 372588 78452 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23112 0.0 3.9 372588 78452 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23113 0.0 3.9 372588 78452 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23117 0.0 3.9 372588 78452 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23118 0.0 3.9 372588 78452 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23122 0.0 3.9 372588 78452 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23125 0.0 3.9 372588 78452 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23128 0.0 3.9 372588 78452 ? S 04:54 0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu 23142 1.3 4.1 373364 82908 ? S 04:54 0:04 airflow scheduler -- DagFileProcessorManager
postgres 23179 0.0 0.9 322380 18120 ? Ss 04:54 0:00 postgres: 10/main: airflow airflow 127.0.0.1(53564) idle
ubuntu 23193 0.4 3.1 217464 63712 ? S 04:54 0:01 gunicorn: master [airflow-webserver]
ubuntu 26167 1.3 5.9 494876 119428 ? Sl 04:58 0:01 [ready] gunicorn: worker [airflow-webserver]
postgres 26181 0.0 0.8 321596 16640 ? Ss 04:58 0:00 postgres: 10/main: airflow airflow 127.0.0.1(54584) idle
ubuntu 26545 1.8 5.9 494876 119428 ? Sl 04:58 0:01 [ready] gunicorn: worker [airflow-webserver]
postgres 26559 0.0 0.8 321596 16640 ? Ss 04:58 0:00 postgres: 10/main: airflow airflow 127.0.0.1(54714) idle
ubuntu 26910 3.6 5.9 494876 119428 ? Sl 04:59 0:01 [ready] gunicorn: worker [airflow-webserver]
postgres 26924 0.0 0.8 321596 16640 ? Ss 04:59 0:00 postgres: 10/main: airflow airflow 127.0.0.1(54840) idle
ubuntu 27287 14.1 5.9 494876 119428 ? Sl 04:59 0:01 [ready] gunicorn: worker [airflow-webserver]
postgres 27301 0.0 0.8 321596 16640 ? Ss 04:59 0:00 postgres: 10/main: airflow airflow 127.0.0.1(54966) idle
ubuntu 27411 0.0 0.0 0 0 ? Z 05:00 0:00 [airflow schedul] <defunct>
ubuntu 27414 0.0 0.0 0 0 ? Z 05:00 0:00 [airflow schedul] <defunct>
ubuntu 27423 0.0 0.1 40268 3884 pts/2 R+ 05:00 0:00 ps -aux --sort start_time
Upvotes: 0
Views: 1188
Answers (1)
Shahbaz Ali
Reputation: 1410
Possible duplicate of following question!
Running `airflow scheduler` launches 33 scheduler processes
Having theses many process is default behavior but you can change it from airflow.cfg by changing parallelism value.
Upvotes: 1
Related Questions
- Apache Airflow error run id already exists
- Apache Airflow keeps repeating DAG
- Airflow 2.0 Issues : Too many airflow supervisor tasks
- Airflow task: OSError: [Errno 23] Too many open files in system
- Airflow too many connections as a default
- Airflow run tasks in parallel
- Running `airflow scheduler` launches 33 scheduler processes
- Run parallel tasks in Apache Airflow
- Running more than 32 concurrent tasks in Apache Airflow
- Running airflow worker gives error: Address already in use