CODEWITHSUNDEEP
linuxdockerubuntuairflow
Shadow Walker
Shadow Walker

Reputation: 1199

How to solve 'no space left on device' error when running airflow on docker

I'm running Airflow on docker, and the scheduler container stops running after some time with the following error

Error response from daemon: Cannot restart container airflow-scheduler_1: mkdir /var/snap/docker/common/var-lib-docker/overlay2/3355a6abc0c5f0bde0eba9026fd04a9d627562d6c6f069c4a564c5fe7738721f/merged: no space left on device

I have used

docker system prune --volumes

but this is effective for a short period of time before the container stops running again and displaying the same error.

What I'm I missing?

Upvotes: 2

Views: 3094

Answers (1)

Thom Bedford
Thom Bedford

Reputation: 387

You've got a few options:

  1. Increase the space on the docker instance itself
  2. Have a repeating script or job that prunes the volumes
  3. Map your DAGs or logs folder in the docker instance to a non-docker mount point

The third option is what we do, all our logs and code files are held outside the docker instances, so they don't get filled up unnecessarily.

We've found that the logs can sometimes be very verbose, so you might also want to look at filtering your logs or adding a log lifecycle.

Upvotes: 4

Related Questions