CODEWITHSUNDEEP
kubernetesrabbitmqpersistencepersistent-storage
JoeMjr2
JoeMjr2

Reputation: 3944

Make RabbitMQ durable/persistent queues survive Kubernetes pod restart

Our application uses RabbitMQ with only a single node. It is run in a single Kubernetes pod.

We use durable/persistent queues, but any time that our cloud instance is brought down and back up, and the RabbitMQ pod is restarted, our existing durable/persistent queues are gone.

At first, I though that it was an issue with the volume that the queues were stored on not being persistent, but that turned out not to be the case.

It appears that the queue data is stored in /var/lib/rabbitmq/mnesia/<user@hostname>. Since the pod's hostname changes each time, it creates a new set of data for the new hostname and loses access to the previously persisted queue. I have many sets of files built up in the mnesia folder, all from previous restarts.

How can I prevent this behavior?

The closest answer that I could find is in this question, but if I'm reading it correctly, this would only work if you have multiple nodes in a cluster simultaneously, sharing queue data. I'm not sure it would work with a single node. Or would it?

Upvotes: 7

Views: 5158

Answers (3)

merrycoder
merrycoder

Reputation: 121

What helped in our case was to set hostname: <static-host-value>

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 1
  ...
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      ...
      containers:
      - name: rabbitmq
        image: rabbitmq:3-management
        ...
      hostname: rmq-host

Upvotes: 3

Rob G
Rob G

Reputation: 3526

I ran into this issue myself and the quickest way I found was to specify an environment variable RABBITMQ_NODENAME = "yourapplicationsqueuename" and making sure I only had 1 replica for my pod.

Upvotes: 0

mdaniel
mdaniel

Reputation: 33231

How can I prevent this behavior?

By using a StatefulSet as is intended for the case where Pods have persistent data that is associated with their "identity." The Helm chart is a fine place to start reading, even if you don't end up using it.

Upvotes: 0

Related Questions