CODEWITHSUNDEEP
postgresqldockerdocker-composeairflowpsycopg2
Eli Sander
Eli Sander

Reputation: 1248

Why can't my airflow webserver initialize properly?

When I create a dummy DAG following the Apache guide to airflow in Docker and run docker-compose up, the webserver container repeatedly fails and restarts with errors that look like this:

webserver_1  | [2021-07-26 18:44:42,905] {cli_action_loggers.py:105} WARNING - Failed to log action with (psycopg2.errors.UndefinedTable) relation "log" does not exist
webserver_1  | LINE 1: INSERT INTO log (dttm, dag_id, task_id, event, execution_dat...

This error suggests to me that the webserver is unable to initialize the airflow db. This dummy DAG uses Airflow 2 and CeleryExecutor. I have run into the same issue following this example using Airflow 2 and LocalExecutor, as well as my team's current setup, which uses Airflow 1 and CeleryExecutor.

What confuses me is that my teammates can run DAGs based on our docker-compose without issue (and presumably, others can run Apache's official quickstart successfully). So something is wrong with my machine (likely related to Postgres) that is somehow affecting my Docker images. What could cause this error across several different docker-compose airflow setups?

What I have tried so far:

I am on Docker version 20.10.7, and docker-compose version 1.29.2.

Upvotes: 2

Views: 3789

Answers (2)

Eli Sander
Eli Sander

Reputation: 1248

I was able to solve this problem by completely cleaning my computer of Docker artifacts using docker system prune. I ran the following commands:

docker system prune --all
docker system prune --all --volumes

My previous attempts to clear Docker artifacts from my computer had been incomplete; killing and removing containers and uninstalling/reinstalling Docker was not sufficient.

Upvotes: 4

Jarek Potiuk
Jarek Potiuk

Reputation: 20097

I believe you did not follow the "Before you begin" steps and increased memory needed for Docker https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#before-you-begin

Default settings for Docker for Mac, does not give you nearly enough memory for Docker to run Airflow.

Upvotes: 1

Related Questions