RabbitMQ fails to boot from docker-compose

Question

Im trying to set up rabbitmq instance from docker-compose command.

My docker compose yaml

version: '3.8'

services:
  rabbitmq:
    image: rabbitmq:3-management
    hostname: rabbit
    container_name: 'rabbitmq'
    volumes:
      - ./etc/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
      - ./data:/var/lib/rabbitmq/mnesia/rabbit@rabbit
      - ./logs:/var/log/rabbitmq/log
      - ./etc/ssl/CERT_LAB_CA.pem:/etc/rabbitmq/ssl/cacert.pem
      - ./etc/ssl/CERT_LAB_RABBITMQ.pem:/etc/rabbitmq/ssl/cert.pem
      - ./etc/ssl/KEY_LAB_RABBITMQ.pem:/etc/rabbitmq/ssl/key.pem
    ports:
      - 5672:5672
      - 15672:15672
      - 15671:15671
      - 5671:5671
    environment:
      - RABBITMQ_DEFAULT_USER=secret
      - RABBITMQ_DEFAULT_PASS=secret

When I run docker compose up for the first time, everything works fine. But when I add queues and exchanged(loaded from definitions.json), shut down and remove container and try to docker compose up again, I got this error

2022-09-29 13:32:09.522956+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0>
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0> BOOT FAILED
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0> ===========
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0> Error during startup: {error,
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0>                           {schema_integrity_check_failed,
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0>                               [{table_missing,rabbit_listener}]}}
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0>

BOOT FAILED
===========
Error during startup: {error,
                          {schema_integrity_check_failed,
                              [{table_missing,rabbit_listener}]}}

2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>   crasher:
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     initial call: application_master:init/4
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     pid: <0.228.0>
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     registered_name: []
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     exception exit: {{schema_integrity_check_failed,
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>                          [{table_missing,rabbit_listener}]},
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>                      {rabbit,start,[normal,[]]}}
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>       in function  application_master:init/4 (application_master.erl, line 142)
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     ancestors: [<0.227.0>]
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     message_queue_len: 1
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     messages: [{'EXIT',<0.229.0>,normal}]
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     links: [<0.227.0>,<0.44.0>]
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     dictionary: []
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     trap_exit: true
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     status: running
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     heap_size: 2586
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     stack_size: 28
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>     reductions: 180
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>   neighbours:
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>

And here is my rabbitmq.conf file

listeners.tcp.default = 5672
listeners.ssl.default = 5671
ssl_options.cacertfile = /etc/rabbitmq/ssl/cacert.pem
ssl_options.certfile = /etc/rabbitmq/ssl/cert.pem
ssl_options.keyfile = /etc/rabbitmq/ssl/key.pem
#Generate client cert and uncomment this if client has to provide cert.
#ssl_options.verify = verify_peer
#ssl_options.fail_if_no_peer_cert = true


collect_statistics_interval = 10000

#load_definitions = /path/to/exported/definitions.json
#definitions.skip_if_unchanged = true

management.tcp.port = 15672


management.ssl.port       = 15671
management.ssl.cacertfile = /etc/rabbitmq/ssl/cacert.pem
management.ssl.certfile   = /etc/rabbitmq/ssl/cert.pem
management.ssl.keyfile    = /etc/rabbitmq/ssl/key.pem

management.http_log_dir = /var/log/rabbitmq/http

What am I missing?

Answer 1

If someone has the same problem, but the solution will not be found for a long time: take a closer look at the paths that you specified in the volume bindings.

For example, my docker compose yml that caused the error looked something like this:

version: "3.8"

services:

  rabbitmq:
    image: rabbitmq:3.11-management-alpine
    hostname: rabbitmq
    ports:
        ...
    environment:
      RABBITMQ_DEFAULT_USER: rmuser
      RABBITMQ_DEFAULT_PASS: rmpassword
    volumes:
        - ./.rabbitmq/data/:/var/lib/rabbitmq/
...

Downgrading to version 3.6-management-alpine helped eliminate the error, but later, when I had to go back to version 3.11-management-alpine, I found that the error was caused by an incorrect path inside the container, which is specified in the bindings. For example:

    volumes:
        - ./.rabbitmq/data/:/var/lib/rabbitmq/

need to fix as following:

    volumes:
        - ./.rabbitmq/data/:/var/lib/rabbitmq

That is, remove the slash at the end of the path that leads to the RabbitMQ data inside the container.

Answer 2

I hit this problem and I changed my docker-compose.yml file to use rabbitmq:3.9-management rather than rabbitmq:3-management.

The problem happened for me when I restarted the stack and the rabbitmq image went to 3.11.

Answer 3

Try to substitute ./data:/var/lib/rabbitmq/mnesia/rabbit@rabbit in your config with ./data:/var/lib/rabbitmq.

I had the same error and spent quite time trying to figure out the problem. My configuration was slightly different from yours and looked like this:

rabbitmq:
  image: rabbitmq:3.11.2-management-alpine
  hostname: rabbitmq
  environment:
    RABBITMQ_DEFAULT_USER: tester
    RABBITMQ_DEFAULT_PASS: qwerty
    RABBITMQ_MNESIA_DIR: /my-custom-data-folder-path-inside-container
    RABBITMQ_NODENAME: rabbitmq
  volumes:
    - type: bind
      source: /my-custom-data-folder-path-on-host
      target: /my-custom-data-folder-path-inside-container

I'm not an expert in RabbitMQ, and my idea was just to make RabbitMQ to persist it's database in the /my-custom-data-folder-path-on-host folder on host. And just like in your case on the first run it was able to start successfully, but after container restart I was getting the following error:

BOOT FAILED Error during startup: {error, {schema_integrity_check_failed, [{table_missing,rabbit_listener}]}}

I learned from the documentation is that rabbit_listener is a table inside the Mnesia database that is used by RabbitMQ and that "listeners" are the TCP-listeners that are configured in RabbitMQ to accept client connections.

For RabbitMQ to accept client connections, it needs to bind to one or more interfaces and listen on (protocol-specific) ports. One such interface/port pair is called a listener in RabbitMQ parlance. Listeners are configured using the listeners.tcp.* configuration option(s).

I wanted to dig into the Mnesia database to troubleshoot but not managed to do that without Erlang knowledge. It seems that for some reason on the first run RabbitMQ does not create "rabbit_listener" table, but on subsequent runs requires it.

Finally, I managed to workaround the problem by changing my initial configuration as follows:

service-bus:
  image: rabbitmq:3.11.2-management-alpine
  hostname: rabbitmq
  environment:
    RABBITMQ_DEFAULT_USER: tester
    RABBITMQ_DEFAULT_PASS: qwerty     
    RABBITMQ_NODENAME: rabbitmq
  volumes:
    - type: bind
      source: /my-custom-data-folder-path-on-host
      target: /var/lib/rabbitmq  

Instead of overriding just the RABBITMQ_MNESIA_DIR folder I've overridden the entire /var/lib/rabbitmq. This did the trick and now my RabbitMQ successfully endures restarts.