CODEWITHSUNDEEP
postgresqldockerdocker-composeairflow
szeta
szeta

Reputation: 599

Airflow via docker-compose keeps trying to access sqlite although postgres configured

I try to set up a Dockerized airflow instance, but whatever I do (so far..) it keeps trying to access some sqlite3 database where I do not know where the instruction comes from. I point to the Postgres instance everywhere (deemed) possible through AIRFLOW__CORE__SQL_ALCHEMY_CONN, and even AIRFLOW_CONN_METADATA_DB.

A typical error message when starting up is like:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: job

Full docker-compose.yml:

version: '3'
x-airflow-common:
  &airflow-common
  image: apache/airflow:2.0.0
  environment:
    - AIRFLOW__CORE__EXECUTOR=LocalExecutor
    - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://postgres:postgres@db:9501/airflow
    - AIRFLOW_CONN_METADATA_DB=postgres+psycopg2://postgres:postgres@db:9501/airflow
    - AIRFLOW__CORE__FERNET_KEY=FB0o_zt4e3Ziq3LdUUO7F2Z95cvFFx16hU8jTeR1ASM=
    - AIRFLOW__CORE__LOAD_EXAMPLES=True
    - AIRFLOW__CORE__LOGGING_LEVEL=INFO
  volumes:
    - /home/x/docker/airflow/dags:/opt/airflow/dags
    - /home/x/docker/airflow/airflow-data/logs:/opt/airflow/logs
    - /home/x/docker/airflow/airflow-data/plugins:/opt/airflow/plugins
    - /home/x/docker/airflow/airflow-data/airflow.cfg:/opt/airlfow/airflow.cfg
  depends_on:
    - db

services:
  db:
    image: postgres:12
    #image: postgres:12.1-alpine
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=airflow
      - POSTGRES_PORT=9501
      - POSTGRES_HOST_AUTH_METHOD=trust
    ports:
      - 9501:9501
    command:
      - -p 9501

  airflow-init:
    << : *airflow-common
    container_name: airflow_init
    entrypoint: /bin/bash
    environment:
      - SQL_ALCHEMY_CONN=postgresql://postgres:postgres@db:9501/airflow
      - AIRFLOW_CONN_METADATA_DB=postgres://postgres:postgres@db:9501/airflow
    command:
      - -c
      - airflow users list || ( airflow db init &&
        airflow users create
          --role Admin
          --username airflow
          --password airflow
          --email [email protected]
          --firstname airflow
          --lastname airflow )
    restart: on-failure

  airflow-webserver:
    << : *airflow-common
    command: airflow webserver
    ports:
      - 9500:8080
    container_name: airflow_webserver
    environment:
      - AIRFLOW_USERNAME=airflow
      - AIRFLOW_PASSWORD=airflow
      - SQL_ALCHEMY_CONN=postgresql://postgres:postgres@db:9501/airflow
      - AIRFLOW_CONN_METADATA_DB=postgres://postgres:postgres@db:9501/airflow
    restart: always

  airflow-scheduler:
    << : *airflow-common
    command: airflow scheduler
    container_name: airflow_scheduler
    environment:
      - SQL_ALCHEMY_CONN=postgresql://postgres:postgres@db:9501/airflow
      - AIRFLOW_CONN_METADATA_DB=postgres://postgres:postgres@db:9501/airflow
    restart: always

Upvotes: 1

Views: 1432

Answers (1)

szeta
szeta

Reputation: 599

Solved by following this docker-compose.yaml file:

https://github.com/apache/airflow/blob/master/docs/apache-airflow/start/docker-compose.yaml

And instead of trying to tweak the ports of postgres (and redis) used the "expose" option, which avoids conflicts with other containers on the same host.

So not:

 environment: 
   POSTGRES_PORT: 9501
 ports:
   - 9501:9501

But: run it (internally) with the default ports and do not try to share them external:

 expose: 
   - 5432

Still not sure what was the problem with using the higher ports. It may be some default fallback to sqlite when the configured DB for some reason cannot be connected.

Upvotes: 1

Related Questions