CODEWITHSUNDEEP
amazon-web-servicesairflowmwaa
xavy
xavy

Reputation: 43

AWS MWAA - Airflow - Load Connections

Im considering move Airflow from ECS to MWAA that apparently works well. But from CICD point of view there are some limitations to load connections. Based in documentation we cannot setup connections using command line based in what they say here: https://docs.aws.amazon.com/mwaa/latest/userguide/access-airflow-ui.html#call-mwaa-apis-cli

I know we can do it using UI, but is not the intention. Does anyone knows how to deal with that or if there is any way to do it on automated way?

Many Thanks Xavy

Upvotes: 1

Views: 2236

Answers (2)

hopeIsTheonlyWeapon
hopeIsTheonlyWeapon

Reputation: 567

There are Couple of options here depending on the version of MWAA you are using.

Hope this helps. Please let me know.

Upvotes: 1

Hedi Bejaoui
Hedi Bejaoui

Reputation: 394

You can add a step in the DAG that will programmatically add the connection if it's not already present in the environment:

def add_connection_callable(**kwargs):
    connection: Connection = Connection(
        conn_id="foo",
        conn_type="HTTP",
        host=kwargs['host'],
        port=kwargs['port']
    )
    session: Session = settings.Session
    db_connection: Connection = session.query(Connection) \
        .filter(Connection.conn_id == "foo") \
        .first()
    if db_connection is None:
        logging.info("Adding connection \"foo\"..")
        session.add(connection)
        session.commit()
    else:
        logging.info("Connection \"foo\" already exists.")

add_connection: PythonOperator = PythonOperator(
    task_id="add_connection",
    python_callable=add_connection_callable,
    op_kwargs={
        "host": "{{ dag_run.conf['Host'] }}",
        "port": "{{ dag_run.conf['Port'] }}"
    },
    dag=dag)

Upvotes: 0

Related Questions