CODEWITHSUNDEEP
pythoncondaairflowubuntu-18.04airflow-scheduler
CurtLH
CurtLH

Reputation: 2417

Running airflow within a conda environment and with systemd fails to launch

I would like to install airflow via conda and use systemd to control airflow on Ubuntu. I've been able to successfully install airflow into a conda environment with the following steps, but I have not been able to correctly configure systemd to work with airflow.

Details:

I've seen SO #52310217, but the question remains unresolved.

Here are the steps that I've taken to install and run airflow from the command line:

$ conda create --name airflow -c conda-forge airflow
$ conda activate airflow
$ export AIRFLOW_HOME=~/airflow
$ airflow initdb

$ airflow scheduler
$ airflow webserver

From here, I've created the following files in /etc/systemd/system/:

# /etc/systemd/system/airflow-scheduler.service
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
Environment="PATH=/home/ubuntu/python/envs/airflow/bin"
User=airflow
Group=airflow
Type=simple
ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow scheduler
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/airflow-webserver.service
[Unit]
Description=Airflow webserver daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
Environment="PATH=/home/ubuntu/python/envs/airflow/bin"
User=airflow
Group=airflow
Type=simple
ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow webserver -p 8085 --pid /home/curtis/airflow/airflow-webserver.pid
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target

After creating the above files, I issued the following commands, but nothing happened.

sudo systemctl daemon-reload
sudo systemctl enable airflow-scheduler
sudo systemctl start airflow-scheduler
sudo systemctl enable airflow-webserver
sudo systemctl start airflow-webserver

If I enter the ExecStart command directly in the terminal window, airflow scheduler and airflow webserver start as expected. For this reason, I don't think it's a problem with airflow, but rather something that I'm missing from the systemd configuration.

I'm not sure what to take away from this, but here is the status of airflow-scheduler and airflow-webserver.

[curtis:~/airflow/logs/scheduler] [airflow] 4 $ sudo systemctl status airflow-scheduler
● airflow-scheduler.service - Airflow scheduler daemon
   Loaded: loaded (/etc/systemd/system/airflow-scheduler.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Mon 2019-08-19 22:36:35 PDT; 3s ago
  Process: 5623 ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow scheduler (code=exited, status=217/USER)
 Main PID: 5623 (code=exited, status=217/USER)

[curtis:~/airflow/logs/scheduler] [airflow] 3 $ sudo systemctl status airflow-webserver
● airflow-webserver.service - Airflow webserver daemon
   Loaded: loaded (/etc/systemd/system/airflow-webserver.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Mon 2019-08-19 22:36:46 PDT; 895ms ago
  Process: 5805 ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow webserver -p 8085 --pid /home/curtis/airflow/airflow-webserver
 Main PID: 5805 (code=exited, status=217/USER)

Upvotes: 3

Views: 3581

Answers (1)

CurtLH
CurtLH

Reputation: 2417

I figured out the issue and wanted to post this here for anyone else that comes across this same issue.

  • The User should be your current username, or the username in which Airflow is installed. In my case that was "curtis"
  • The Group should be your current group, or the group of the user in which Airflow is installed. In my case that was "curtis"
  • The Environment is the path to your conda environment. In my case, that was PATH=/home/curtis/miniconda3/envs/airflow/bin

I've included my corrected systemd files for your reference. 

# /etc/systemd/system/airflow-scheduler.service
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
Environment="PATH=/home/curtis/miniconda3/envs/airflow/bin"
User=curtis
Group=curtis
Type=simple
ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow scheduler
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/airflow-webserver.service
[Unit]
Description=Airflow webserver daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
Environment="PATH=/home/curtis/miniconda3/envs/airflow/bin"
User=curtis
Group=curtis
Type=simple
ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow webserver -p 8085 --pid /home/curtis/airflow/airflow-webserver.pid
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Upvotes: 3

Related Questions