CODEWITHSUNDEEP
amazon-web-servicesamazon-rdsairflow
asur
asur

Reputation: 1879

Enable RDS authentication in Airflow

I have configured Airflow with AWS RDS as backend DB. Now this UI has no authentication. i.e, whenever I hit the URL it shows me the Dags etc. How can I enable RDS authentication in Airflow now? Please suggest.

Airflow Version : 1.10.3

I followed below Link just to make sure I can create some user but I want AirFlow authenticates users using credentials stored in RDS.

https://airflow.apache.org/cli.html#create_user

Error after executing script provided by @Anup:-

[ec2-user@ip-10-123-123-123 airflow]$ python3.7 authenticate.py 
/usr/local/lib/python3.7/site-packages/airflow/configuration.py:214: FutureWarning: The task_runner setting in [core] has the old default value of 'BashTaskRunner'. This value has been changed to 'StandardTaskRunner' in the running config, but please update your config before Apache Airflow 2.0.
  FutureWarning,
/usr/local/lib/python3.7/site-packages/airflow/configuration.py:575: DeprecationWarning: Specifying airflow_home in the config file is deprecated. As you have left it at the default value you should remove the setting from your airflow.cfg and suffer no change in behaviour.
  category=DeprecationWarning,
[2019-08-05 09:35:51,140] {settings.py:182} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=2000, pid=5166
Traceback (most recent call last):
  File "authenticate.py", line 14, in <module>
    engine = create_engine("db+mysql://airflow:airflow1234@abc-def-ghi-airflow.abcdefghijkl.eu-central-1.rds.amazonaws.com:3306/airflow")
  File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/__init__.py", line 443, in create_engine
    return strategy.create(*args, **kwargs)
  File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 61, in create
    entrypoint = u._get_entrypoint()
  File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
    cls = registry.load(name)
  File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 232, in load
    "Can't load plugin: %s:%s" % (self.group, name)
sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:db.mysql

Upvotes: 0

Views: 1159

Answers (1)

Anup
Anup

Reputation: 250

By default, the config file which is under config/airflow.cfg is enabled with web authentication, to disable it, change the boolean value from "True" to "False"

Below is the configuaration to enable,

# Set to true to turn on authentication:
# https://airflow.apache.org/security.html#web-authentication
authenticate = True
auth_backend = airflow.contrib.auth.backends.password_auth

Follow the below steps to create the users,

Get the container id,

docker container ls

Jump onto the contianer bash by it's id,

# With root user
docker exec -it -u root <container id> bash

Execute the follow script under the airflow folder with in the python console

python
import airflow
from airflow import models, settings
from airflow.contrib.auth.backends.password_auth import PasswordUser
from sqlalchemy import create_engine

user = PasswordUser(models.User())
user.username = ''
user.email = ''
user.password = ''

# Make the value true if you want the user to be a administrator
user.superuser = False

engine = create_engine("postgresql://airflow:airflow@postgres:5432/airflow")
session = settings.Session(bind=engine)
session.add(user)
session.commit()
session.close()
exit()

Enter exit and hit enter to come out of the container bash

Here is an airflow-docker boilerplate with localexecutor.

NOTE: I created this boilerplate, since the authentication and the overall docker implementation out there on github(https://github.com/saianupkumarp/airflow-docker) weren't making any sense to me.

Upvotes: 2

Related Questions