CODEWITHSUNDEEP
dockerdocker-composeairflow
Mohamed Yasser
Mohamed Yasser

Reputation: 812

Airflow init in Docker creates empty directories

One of the projects I am working on is using airflow. So, I used airflow's documentation to install airflow with docker compose.

I wanted the dags plugin and logs directories to not be in the same directory as the docker-compose.yaml, so I put them inside another directory called airflow and referenced the directory inside the docker-compose.yaml, like so:

  volumes:
    - ./airflow/dags:/opt/airflow/dags
    - ./airflow/logs:/opt/airflow/logs
    - ./airflow/plugins:/opt/airflow/plugins

Whenever I run the docker-compose, airflow-init service creates the three directories again in the same directory as the docker-compose.yaml (I realized they appear when it starts running). It is not a big problem, since I can just ignore them. They are empty and airflow correctly uses the directories inside the airflow directory. But I wanted to know if there was a way to stop these directories from being created.

Upvotes: 1

Views: 971

Answers (1)

Mohamed Yasser
Mohamed Yasser

Reputation: 812

Thanks to @EDG956 for the help.

There is a volumes field inside the airflow-init service that I completely missed. You can change the volume mount from

volumes:
      - .:/sources

to:

volumes:
      - ./airflow/:/sources

Or change the part of the command field that creates the directories in the /sources volume, like so:

        mkdir -p /sources/airflow/logs /sources/airflow/dags /sources/airflow/plugins
        chown -R "${AIRFLOW_UID}:0" /sources/airflow/{logs,dags,plugins}
        exec /entrypoint airflow version
    # yamllint enable rule:line-length
    environment:

Although as @EDG956 pointed out, the first solution seems to be more efficient.

Upvotes: 1

Related Questions