CODEWITHSUNDEEP
airflowairflow-scheduler
italktothewind
italktothewind

Reputation: 2195

Airflow: connection is not being created when using environment variables

I want to create a Mongo connection (other than default) without using the Airflow UI.

I read from the Airflow documentation:

Connections in Airflow pipelines can be created using environment variables. The environment variable needs to have a prefix of AIRFLOW_CONN_ for Airflow with the value in a URI format to use the connection properly.

When referencing the connection in the Airflow pipeline, the conn_id should be the name of the variable without the prefix. For example, if the conn_id is named postgres_master the environment variable should be named AIRFLOW_CONN_POSTGRES_MASTER (note that the environment variable must be all uppercase).

I tried to apply this when using the Puckel docker image.

This is a docker compose using that image:

version: '2.1'
 services:
    postgres:
        image: postgres:9.6
        environment:
            - POSTGRES_USER=airflow
            - POSTGRES_PASSWORD=airflow
            - POSTGRES_DB=airflow

    webserver:
        image: puckel/docker-airflow:1.10.6
        restart: always
        depends_on:
            - postgres
        environment:
            - LOAD_EX=n
            - EXECUTOR=Local
            - AIRFLOW_CONN_MY_MONGO=mongodb://mongo:27017
        volumes:
            - ./src/:/usr/local/airflow/dags
            - ./requirements.txt:/requirements.txt
        ports:
            - "8080:8080"
        command: webserver
        healthcheck:
            test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
            interval: 30s
            timeout: 30s
            retries: 3

Note the line AIRFLOW_CONN_MY_MONGO=mongodb://mongo:27017 where I'm passing the environment variable as the Airflow documentation suggests.

Problem here is that there is no my_mongo connection created when I'm listing the connections in the UI.

Any advice? Thanks!

Upvotes: 4

Views: 5184

Answers (2)

kaxil
kaxil

Reputation: 18824

The connection won't be listed in the UI when you create it with environment variable.

Reason:

  • Airflow supports the creation of connections via Environment variable for ad-hoc jobs in the DAGs
  • The connection in the UI are actually saved in the DB and retrieved from it. The ones created by Env vars are not stored in DB

How do I test my connection?

  • Create a sample DAG and use your connection to run a sample job. It should work fine.

Upvotes: 13

italktothewind
italktothewind

Reputation: 2195

I read a Puckel issue where they mention that the connection is created, but is not showed in the UI. I tested it and in fact the connection works when used in a DAG.

Upvotes: 1

Related Questions