RabbitMQ Cluster Kubernetes Operator on minikube

Question

I'm trying to set up RabbitMQ on Minikube using the RabbitMQ Cluster Operator:

When I try to attach a persistent volume, I get the following error:

$ kubectl logs -f rabbitmq-rabbitmq-server-0

Configuring logger redirection
20:04:40.081 [warning] Failed to write PID file "/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-rabbitmq-server-0.rabbitmq-rabbitmq-headless.default.pid": permission denied
20:04:40.264 [error] Failed to create Ra data directory at '/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-rabbitmq-server-0.rabbitmq-rabbitmq-headless.default/quorum/rabbit@rabbitmq-rabbitmq-server-0.rabbitmq-rabbitmq-headless.default', file system operation error: enoent
20:04:40.265 [error] Supervisor ra_sup had child ra_system_sup started with ra_system_sup:start_link() at undefined exit with reason {error,"Ra could not create its data directory. See the log for details."} in context start_error
20:04:40.266 [error] CRASH REPORT Process <0.247.0> with 0 neighbours exited with reason: {error,"Ra could not create its data directory. See the log for details."} in ra_system_sup:init/1 line 43
20:04:40.267 [error] CRASH REPORT Process <0.241.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,ra_system_sup,{error,"Ra could not create its data directory. See the log for details."}}},{ra_app,start,[normal,[]]}} in application_master:init/4 line 138
{"Kernel pid terminated",application_controller,"{application_start_failure,ra,{{shutdown,{failed_to_start_child,ra_system_sup,{error,\"Ra could not create its data directory. See the log for details.\"}}},{ra_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,ra,{{shutdown,{failed_to_start_child,ra_system_sup,{error,"Ra could not create its data directory. See the log for details."}

Crash dump is being written to: erl_crash.dump...

The issue is that RabbitMQ is not able to set up its data files in the data directory /var/lib/rabbitmq/mnesia due to a lack of permission.

My initial guess was that I needed to specify the data directory as a volumeMount, but that doesn't seem to be configurable according to the documentation.

RabbitMQ's troubleshooting documentation on persistence results in a 404.

I tried to find other resources online with the same problem but none of them were using the RabbitMQ Cluster Operator. I plan on following that route if I'm not able to find a solution but I really would like to solve this issue somehow.

Does anyone have any ideas?

The setup that I have is as follows:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq
spec:
  replicas: 1
  service:
    type: NodePort
  persistence:
    storageClassName: local-storage
    storage: 20Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rabbitmq-pvc
spec:
  storageClassName: local-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: rabbitmq-pv
spec:
  storageClassName: local-storage
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 20Gi
  hostPath:
    path: /mnt/app/rabbitmq
    type: DirectoryOrCreate

To reproduce this issue on minikube:

1. Install the rabbitmq operator:

kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"

2. Apply the manifest file above

kubectl apply -f rabbitmq.yml

3. Running kubectl get po displays a pod named rabbitmq-rabbitmq-server-0.

4. Running kubectl logs -f rabbitmq-rabbitmq-server-0 to view the logs displays the above error.

Answer 1

I found the answer for this issue. It happen when there is few nodes in cluster. Solution is to add securityContext: {}

https://github.com/rabbitmq/rabbitmq-website/blob/3ee8e72a7c4fe52e323ba1039eecbf3a67c554f7/site/kubernetes/operator/using-on-openshift.md#arbitrary-user-ids

Answer 2

I had this issue in a live version and minikube does not allow to run SSH command then.So what I did was run chmod to the my hostpath provisioner and recreated my rabbitmq-cluster

chmod 777 /tmp/hostpath-provisioner/default/*

Answer 3

I had the same issue when deploying RabbitMQ in kubernetes inside Vagrant (not minikube though). I was using this setup. I tried running sudo chmod g+w /mnt/app/rabbitmq/ but had no luck... Eventually gave up and ended up running minikube inside vagrant using this box and everything worked perfectly fine out of the box! Didn't have to do anything special... Not event manually creating the persistent volume...

inside my nodes

Answer 4

As I alread suggested in comments, you can solve it running:

minikube ssh -- sudo chmod g+w /mnt/app/rabbitmq/ 

---

Answering to your question:

Is there a way I can add that to my manifest file rather than having to do it manually?

You can override the rabbitmq statefulset manifest fields to change last line in initContainer command script from chgrp 999 /var/lib/rabbitmq/mnesia/ to this: chown 999:999 /var/lib/rabbitmq/mnesia/.

Complete RabbitmqCluster yaml manifest looks like following:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq
spec:
  replicas: 1
  service:
    type: NodePort
  persistence:
    storageClassName: local-storage
    storage: 20Gi
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers: []
            initContainers:
            - name: setup-container
              command:
              - sh
              - -c
              - cp /tmp/rabbitmq/rabbitmq.conf /etc/rabbitmq/rabbitmq.conf && chown 999:999
                /etc/rabbitmq/rabbitmq.conf && echo '' >> /etc/rabbitmq/rabbitmq.conf ; cp /tmp/rabbitmq/advanced.config
                /etc/rabbitmq/advanced.config && chown 999:999 /etc/rabbitmq/advanced.config
                ; cp /tmp/rabbitmq/rabbitmq-env.conf /etc/rabbitmq/rabbitmq-env.conf && chown
                999:999 /etc/rabbitmq/rabbitmq-env.conf ; cp /tmp/erlang-cookie-secret/.erlang.cookie
                /var/lib/rabbitmq/.erlang.cookie && chown 999:999 /var/lib/rabbitmq/.erlang.cookie
                && chmod 600 /var/lib/rabbitmq/.erlang.cookie ; cp /tmp/rabbitmq-plugins/enabled_plugins
                /etc/rabbitmq/enabled_plugins && chown 999:999 /etc/rabbitmq/enabled_plugins
                ; chown 999:999 /var/lib/rabbitmq/mnesia/ # <- CHANGED THIS LINE