docker-compose keeps previous environment variables after restart

Question

I am an experienced software developer, but fairly new to docker.

I am trying to build a development environment for Magento 2.4 using the bitnami/magento base image (https://hub.docker.com/r/bitnami/magento). When I first downloaded the docker-compose.yml and ran it, everything worked fine right away.

Note: This is not a Magento question. I think the specific container used is secondary to my problem. It is rather a docker/docker-compose on Mac question.

The original docker-compose.yml file I used:

version: '2'
services:
  mariadb:
    image: docker.io/bitnami/mariadb:10.3
    environment:
      - ALLOW_EMPTY_PASSWORD=yes
      - MARIADB_USER=bn_magento
      - MARIADB_DATABASE=bitnami_magento
    volumes:
      - 'mariadb_data:/bitnami/mariadb'
  magento:
    image: docker.io/bitnami/magento:2
    ports:
      - '80:8080'
      - '443:8443'
    environment:
      - MAGENTO_HOST=localhost
      - MAGENTO_DATABASE_HOST=mariadb
      - MAGENTO_DATABASE_PORT_NUMBER=3306
      - MAGENTO_DATABASE_USER=bn_magento
      - MAGENTO_DATABASE_NAME=bitnami_magento
      - ELASTICSEARCH_HOST=elasticsearch
      - ELASTICSEARCH_PORT_NUMBER=9200
      - ALLOW_EMPTY_PASSWORD=yes
    volumes:
      - 'magento_data:/bitnami/magento'
    depends_on:
      - mariadb
      - elasticsearch
  elasticsearch:
    image: docker.io/bitnami/elasticsearch:6
    volumes:
      - 'elasticsearch_data:/bitnami/elasticsearch/data'
volumes:
  mariadb_data:
    driver: local
  magento_data:
    driver: local
  elasticsearch_data:
    driver: local

But then I wanted to make adjustments, such as configuring the local path where the source code was mapped via a volume or the Magento password. For that, I reconfigured the container to be a root container by adding user: root (see below).

What I did was create a .env file where I defined a few variables and then applied them in the docker-compose file, like this:

.env file:

ENV_MAGENTO_PASS=admin

Adapted yml file (added ENV_MAGENTO_PASS as an example):

  magento:
    image: docker.io/bitnami/magento:2
    user: root
    ports:
      - '80:8080'
      - '443:8443'
    environment:
      - MAGENTO_HOST=localhost
      - MAGENTO_DATABASE_HOST=mariadb
      - MAGENTO_DATABASE_PORT_NUMBER=3306
      - MAGENTO_DATABASE_USER=bn_magento
      - MAGENTO_DATABASE_NAME=bitnami_magento
      - MAGENTO_PASSWORD=${ENV_MAGENTO_PASS}
      - ELASTICSEARCH_HOST=elasticsearch
      - ELASTICSEARCH_PORT_NUMBER=9200
      - ALLOW_EMPTY_PASSWORD=yes
    volumes:
      - /my/local/magento/devpath:/bitnami/magento
    depends_on:
      - mariadb
      - elasticsearch

I also meddled with the ALLOW_EMPTY_PASSWORD options in both locations and tried to change the volume mapping through a variable, but realized - much later - that environment variables can only be applied on the value (right) side in the yml file, not on the key side.

The point is, in order to get all this to work, I had to destroy and recreate everything many times. I used a shell script for this (commented lines are what I have tried as well):

docker pull bitnami/magento:2.4.2
#curl -sSL https://raw.githubusercontent.com/bitnami/bitnami-docker-magento/master/docker-compose.yml > docker-compose.yml

#docker builder prune
#docker system prune -a

#docker-compose down
docker-compose rm -f -s -v
#docker rm -f magento2_magento_1 magento2_elasticsearch_1 magento2_mariadb_1
#docker volume rm $(docker volume ls -q)
docker rmi -f $(docker images | grep bitnami/ | tr -s ' ' | cut -d ' ' -f 3)
docker-compose build --no-cache
docker-compose up -d --force-recreate

As you can see, I usually tried to wipe everything, containers and images, and recreate from scratch. Nevertheless, I couldn't get my simple changes to work and at some point I realized it was because the environment variables weren't catching on. Or sometimes, when I reverted my changes, some previously declared environment variables were still present (or absent) so that the desired information wasn't getting through.

I tried everything as you can see in the scripts, even deleting all caches and images but sometimes the old environment variables would still be there!

For example, with the configuration detailed above, I am getting this error message after running everything and doing a docker run on the magento image (because the container exited):

michaelheumann@Michaels-MacBook-Pro magento2 % docker run 0298768ce79e
mariadb 20:46:50.12
mariadb 20:46:50.12 Welcome to the Bitnami mariadb container
mariadb 20:46:50.12 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb
mariadb 20:46:50.12 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues
mariadb 20:46:50.12
mariadb 20:46:50.13 INFO  ==> ** Starting MariaDB setup **
mariadb 20:46:50.14 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 20:46:50.14 ERROR ==> The MARIADB_ROOT_PASSWORD environment variable is empty or not set. Set the environment variable ALLOW_EMPTY_PASSWORD=yes to allow the container to be started with blank passwords. This is recommended only for development.

But ALLOW_EMPTY_PASSWORD is set!!

Has anyone experienced an issue like this? I work on MacOS Big Sur and I understand Docker on Mac creates some kind of hidden virtual machine to represent Docker containers. Could it be that this is the reason why the environment is not easily cleared?

Does anyone have a recommendation on how to avoid this kind of problem or an explanation why this happens?

By the way: Is there a way to make my host path for the volume configurable?

Sorry for the long read and thanks for any help.

Answer 1

In the end, the problem with the environment variables was related to my executing docker run on single images instead of docker-compose run, so the messages were really not related.

And the other things were likely a problem with the volume. I ended up using this docker-composer.yml:

version: '2'
services:
  mariadb:
    image: docker.io/bitnami/mariadb:10.3
    environment:
      - ALLOW_EMPTY_PASSWORD=yes
      - MARIADB_USER=bn_magento
      - MARIADB_DATABASE=bitnami_magento
    volumes:
      - 'mariadb_data:/bitnami/mariadb'
  magento:
    image: docker.io/bitnami/magento:2
    user: root
    ports:
      - '80:8080'
      - '443:8443'
    environment:
      - PHP_MEMORY_LIMIT=512m
      - MAGENTO_HOST=localhost
      - MAGENTO_DATABASE_HOST=mariadb
      - MAGENTO_DATABASE_PORT_NUMBER=3306
      - MAGENTO_DATABASE_USER=bn_magento
      - MAGENTO_DATABASE_NAME=bitnami_magento
      - ELASTICSEARCH_HOST=elasticsearch
      - ELASTICSEARCH_PORT_NUMBER=9200
      - ALLOW_EMPTY_PASSWORD=yes
    volumes:
      - 'magento_data:/bitnami/magento'
    depends_on:
      - mariadb
      - elasticsearch
  elasticsearch:
    image: docker.io/bitnami/elasticsearch:6
    volumes:
      - 'elasticsearch_data:/bitnami/elasticsearch/data'
volumes:
  mariadb_data:
    driver: local
  magento_data:
    driver: local
    driver_opts:
      type: none
      device: ${ENV_MAGENTO_LOCAL_PATH}
      o: bind
  elasticsearch_data:
    driver: local

with ENV_MAGENTO_LOCAL_PATH declared in my .env file.

I needed to remove all volumes prior to rebuilding everything and then it finally worked.

Just in case something like this ever happens to someone else.