CODEWITHSUNDEEP
airflow
Paniz Asghari
Paniz Asghari

Reputation: 305

Airflow webserver causes OS to crash

I am running Airflow on an Ubuntu VM. Here is my config file values for Airflow:

plugins_folder = /home/helia/airflow/utils
dags_folder = /home/helia/airflow/dags

Here are my environment variables set accordingly:

>>echo $PYTHONPATH
/home/helia/airflow:/home/helia/airflow/utils
>>>echo $AIRFLOW_HOME
/home/helia/airflow

If I set my plugins folder path in the cfg file to some non-existing file, the webserver successfully starts and produces the errors fast. Howver, if I change the cfg file plugins path to the correct path, the OS crashes every time with 100% of CPU, swap space and Memory usage. I tried to increase number of processors and memory size but nothing got better.

I am using:
Airflow 1.10.12
Python 3.8
UBUNTU 20.1

Upvotes: 0

Views: 593

Answers (1)

Jarek Potiuk
Jarek Potiuk

Reputation: 20077

I am afraid it's your custom plugin that crash the OS. There is not much that anyone could do without knowing and having access to your plugins. You have to revert to those who wrote those plugins to debug them. –

Plugins are loaded by webserver (one of the features of plugins is that they can extend the UI of webserver). Depending on your deployment the "plugin" folder should be shared with the webserver (usually via shared filesystem) –

Upvotes: 1

Related Questions