CODEWITHSUNDEEP
mongodbkubernetes
Mike Me
Mike Me

Reputation: 2392

Change password in mongodb deployed on kubernetes

I am unable to change the password of an existing user from MongoDB deployed on k8s, unless I am deleting the database and then recreating it again with the new password.

How can I change the password using the yaml for the mongo stateful object without deleting the db?

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo-db-statefulset
  namespace: development
spec:
  serviceName: mongo-svc
  replicas: 1
  selector:
    matchLabels:
      component: mongo
  template:
    metadata:
      labels:
        component: mongo
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo:4.0.4
          volumeMounts:
            - mountPath: /data/db
              name: volume 
          ports:
            - containerPort: 27017
          env:
            - name: MONGO_INITDB_ROOT_USERNAME
              value: admin
            - name: MONGO_INITDB_ROOT_PASSWORD
             # from secrets
              value: password
            - name: MONGO_INITDB_DATABASE
              value: admin
      volumes:
        - name: volume
          persistentVolumeClaim:
            claimName: database-persistent-volume-claim

Upvotes: 4

Views: 2450

Answers (3)

Marquee
Marquee

Reputation: 1826

If using the Bitnami MongoDB helm chart, then updating secrets and restarting does not work.

The only way we have found is to either have to update the password using typical MongoDB capabilities, or blow away the Persistent Volume Claim and create it anew (which means you also have to reload your data).

Since reloading the data is a pain, we just created a procedure to update the credentials:

  1. Update the credentials in your k8s secret.
    • Note: we use akv2k8s to manage credentials via the Azure Key Vault. Highly recommended if you're running on AKS!
  2. Run the MongoDB updateUser command: 
    mongosh --host <host> --port <port> --username root --authenticationDatabase admin <database> --eval 'db.changeUserPassword("<user>", "<new-password>")'
    • Note that you will be prompted to enter the root user password. It can also be passed with --password <password> inline but that is not recommended (typical security risks associated with plain text passwords on CLI).
  3. Restart any apps consuming the database / using the credential (they use the same key vault entry with akv2k8s). 
    kubectl rollout restart --namespace <namespace> deployment/<deployment>

Upvotes: 0

Simu Liviu
Simu Liviu

Reputation: 41

Example of command for @kane, I don't have reputation to add as a comment: kubectl -n namespace rollout restart deployment/name_of_deployment

Edit: I changed the password in the secret, yet after kubectl rollout restart mongo somehow uses the old password, not sure how. Edit2: If I connect to mongo pod, variable MONGO_INITDB_ROOT_PASSWORD shows the new password, yet connecting to mongo db works only with the old one Edit3: What I ended up doing was: 1. Keep the secret with new value, not sure if it even matters at this point. 2. Login to pod and run this 2 commands: db = db.getSiblingDB('admin') followed by db.changeUserPassword("root", "NEWPASSWORD") 3. Just to test this is persistent, I deleted the deployment and created it again and the new password is working, old one not anymore

Upvotes: 0

Jakub
Jakub

Reputation: 8830

If I understand your issue correctly:

  • You have secret with your password as environment variable, and pod has access to the secret data through a Volume
  • You changed the secret password, but it's not getting picked up by a pod without a restart.

According to documentation:

Environment variables are not updated after a secret update, so if If a container already consumes a Secret in an environment variable, a Secret update will not be seen by the container unless it is restarted. There are third party solutions for triggering restarts when secrets change.

This is a known issue. You can read more about it in this github issue.

So after you change the secret password you have to restart your pod to update this value, you don't have to delete it.

As mentioned in documentation there are third party tools for triggering restart when secrets change, one of them is Reloader.

Reloader can watch changes in ConfigMap and Secret and do rolling upgrades on Pods with their associated DeploymentConfigs, Deployments, Daemonsets and Statefulsets.

The quick way to restart deployment would be to use kubectl rollout restart, which performs a step by step shutdown and restarts each container in your deployment or statefulset.

If you change the password in your secret and use kubectl rollout restart the new password should work.

Upvotes: 4

Related Questions