Miaucl
Reputation: 364
CrashLoopBackOff on postgresql bitnami helm chart
I know there have been already a lot of questions about this, and I read already most of them, but my problem does not seem to fit them.
I am running a postgresql from bitnami using a helm chart as described below. A clean setup is no problem and everything starts fine. But after some time, until now I could not find any pattern, the pod goes into CrashLoopBackOff and I cannot recover it whatever I try!
Helm uninstall/install does not fix the problem. The PVs seem to be the problem, but I do not know why. And I do not get any error message, which is the weird and scary part of it.
I use a minikube to run the k8s and helm v3.
Here are the definitions and logs:
# Source: aposphere/charts/sessiondb/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: sessiondb
labels:
app.kubernetes.io/name: sessiondb
helm.sh/chart: sessiondb-10.9.6
app.kubernetes.io/instance: asdf
app.kubernetes.io/managed-by: Helm
annotations:
namespace: default
spec:
type: ClusterIP
ports:
- name: tcp-postgresql
port: 5432
targetPort: tcp-postgresql
selector:
app.kubernetes.io/name: sessiondb
app.kubernetes.io/instance: asdf
role: primary
---
# Source: aposphere/charts/sessiondb/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: sessiondb
labels:
app.kubernetes.io/name: sessiondb
helm.sh/chart: sessiondb-10.9.6
app.kubernetes.io/instance: asdf
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: primary
annotations:
namespace: default
spec:
serviceName: sessiondb-headless
replicas: 1
updateStrategy:
type: RollingUpdate
selector:
matchLabels:
app.kubernetes.io/name: sessiondb
app.kubernetes.io/instance: asdf
role: primary
template:
metadata:
name: sessiondb
labels:
app.kubernetes.io/name: sessiondb
helm.sh/chart: sessiondb-10.9.6
app.kubernetes.io/instance: asdf
app.kubernetes.io/managed-by: Helm
role: primary
app.kubernetes.io/component: primary
spec:
affinity:
podAffinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: sessiondb
app.kubernetes.io/instance: asdf
app.kubernetes.io/component: primary
namespaces:
- "default"
topologyKey: kubernetes.io/hostname
weight: 1
nodeAffinity:
securityContext:
fsGroup: 1001
automountServiceAccountToken: false
containers:
- name: sessiondb
image: docker.io/bitnami/postgresql:11.13.0-debian-10-r33
imagePullPolicy: "IfNotPresent"
resources:
requests:
cpu: 250m
memory: 256Mi
securityContext:
runAsUser: 1001
env:
- name: BITNAMI_DEBUG
value: "false"
- name: POSTGRESQL_PORT_NUMBER
value: "5432"
- name: POSTGRESQL_VOLUME_DIR
value: "/bitnami/postgresql"
- name: PGDATA
value: "/bitnami/postgresql/data"
- name: POSTGRES_USER
value: "postgres"
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgresql-root-password
key: postgresql-password
- name: POSTGRES_DB
value: "session"
- name: POSTGRESQL_ENABLE_LDAP
value: "no"
- name: POSTGRESQL_ENABLE_TLS
value: "no"
- name: POSTGRESQL_LOG_HOSTNAME
value: "false"
- name: POSTGRESQL_LOG_CONNECTIONS
value: "false"
- name: POSTGRESQL_LOG_DISCONNECTIONS
value: "false"
- name: POSTGRESQL_PGAUDIT_LOG_CATALOG
value: "off"
- name: POSTGRESQL_CLIENT_MIN_MESSAGES
value: "error"
- name: POSTGRESQL_SHARED_PRELOAD_LIBRARIES
value: "pgaudit"
ports:
- name: tcp-postgresql
containerPort: 5432
livenessProbe:
exec:
command:
- /bin/sh
- -c
- exec pg_isready -U "postgres" -d "dbname=session" -h 127.0.0.1 -p 5432
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 6
readinessProbe:
exec:
command:
- /bin/sh
- -c
- -e
- |
exec pg_isready -U "postgres" -d "dbname=session" -h 127.0.0.1 -p 5432
[ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 6
volumeMounts:
- name: custom-init-scripts
mountPath: /docker-entrypoint-initdb.d/
- name: dshm
mountPath: /dev/shm
- name: data
mountPath: /bitnami/postgresql
subPath:
volumes:
- name: custom-init-scripts
configMap:
name: sessiondb-scheme
- name: dshm
emptyDir:
medium: Memory
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "8Gi"
Logs of the container:
% kubectl logs sessiondb-0
postgresql 10:09:01.48
postgresql 10:09:01.48 Welcome to the Bitnami postgresql container
postgresql 10:09:01.49 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
postgresql 10:09:01.49 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
postgresql 10:09:01.49
postgresql 10:09:01.50 DEBUG ==> Configuring libnss_wrapper...
postgresql 10:09:01.51 INFO ==> ** Starting PostgreSQL setup **
postgresql 10:09:01.54 INFO ==> Validating settings in POSTGRESQL_* env vars..
postgresql 10:09:01.55 INFO ==> Loading custom pre-init scripts...
postgresql 10:09:01.55 INFO ==> Initializing PostgreSQL database...
postgresql 10:09:01.56 DEBUG ==> Ensuring expected directories/files exist...
postgresql 10:09:01.57 INFO ==> pg_hba.conf file not detected. Generating it...
postgresql 10:09:01.58 INFO ==> Generating local authentication configuration
postgresql 10:09:01.58 INFO ==> Deploying PostgreSQL with persisted data...
postgresql 10:09:01.60 INFO ==> Configuring replication parameters
postgresql 10:09:01.65 INFO ==> Configuring fsync
postgresql 10:09:01.71 INFO ==> Loading custom scripts...
postgresql 10:09:01.72 INFO ==> Loading user's custom files from /docker-entrypoint-initdb.d ...
postgresql 10:09:01.72 INFO ==> Starting PostgreSQL in background...
pg_ctl: directory "/bitnami/postgresql/data" is not a database cluster directory
Afterwards the container terminates, no more logs!
Logs of the init container:
% kubectl logs sessiondb-0 init-chmod-data
+ chown 1001:1001 /bitnami/postgresql
+ mkdir -p /bitnami/postgresql/data
+ chmod 700 /bitnami/postgresql/data
+ find /bitnami/postgresql -mindepth 1 -maxdepth 1 -not -name conf -not -name .snapshot -not -name lost+found
+ xargs chown -R 1001:1001
+ chmod -R 777 /dev/shm
Permissions:
I have no name!@sessiondb-0:/$ stat /bitnami/postgresql/data
File: /bitnami/postgresql/data
Size: 207 Blocks: 0 IO Block: 4096 directory
Device: 10301h/66305d Inode: 712929 Links: 12
Access: (0700/drwx------) Uid: ( 1001/ UNKNOWN) Gid: ( 1001/ UNKNOWN)
Access: 2021-11-10 15:16:13.958633094 +0000
Modify: 2021-11-26 08:40:42.621884636 +0000
Change: 2021-11-26 10:37:47.844490933 +0000
Birth: -
Describe the resources:
Name: sessiondb-0
Namespace: default
Priority: 0
Node: ip-10-0-1-112.eu-central-1.compute.internal/10.0.1.112
Start Time: Fri, 26 Nov 2021 10:40:02 +0100
Labels: app.kubernetes.io/component=primary
app.kubernetes.io/instance=asdf
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=sessiondb
controller-revision-hash=sessiondb-578ddf476b
helm.sh/chart=sessiondb-10.9.6
role=primary
statefulset.kubernetes.io/pod-name=sessiondb-0
Annotations: <none>
Status: Running
IP: 172.17.0.4
IPs:
IP: 172.17.0.4
Controlled By: StatefulSet/sessiondb
Containers:
sessiondb:
Container ID: docker://a94f894687f0813196a94afe88f64723f238eb7d2cb061e4c7ef17354f27dee8
Image: docker.io/bitnami/postgresql:11.13.0-debian-10-r33
Image ID: docker-pullable://bitnami/postgresql@sha256:205e1c5a1d4b56d0d63f6579557652f958e321006c4cb5325d031d40313e4ea2
Port: 5432/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 26 Nov 2021 10:50:45 +0100
Finished: Fri, 26 Nov 2021 10:50:46 +0100
Ready: False
Restart Count: 7
Requests:
cpu: 250m
memory: 256Mi
Liveness: exec [/bin/sh -c exec pg_isready -U "postgres" -d "dbname=session" -h 127.0.0.1 -p 5432] delay=30s timeout=5s period=10s #success=1 #failure=6
Readiness: exec [/bin/sh -c -e exec pg_isready -U "postgres" -d "dbname=session" -h 127.0.0.1 -p 5432
[ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]
] delay=5s timeout=5s period=10s #success=1 #failure=6
Environment:
BITNAMI_DEBUG: false
POSTGRESQL_PORT_NUMBER: 5432
POSTGRESQL_VOLUME_DIR: /bitnami/postgresql
PGDATA: /bitnami/postgresql/data
POSTGRES_USER: postgres
POSTGRES_PASSWORD: <set to the key 'postgresql-password' in secret 'postgresql-root-password'> Optional: false
POSTGRES_DB: session
POSTGRESQL_ENABLE_LDAP: no
POSTGRESQL_ENABLE_TLS: no
POSTGRESQL_LOG_HOSTNAME: false
POSTGRESQL_LOG_CONNECTIONS: false
POSTGRESQL_LOG_DISCONNECTIONS: false
POSTGRESQL_PGAUDIT_LOG_CATALOG: off
POSTGRESQL_CLIENT_MIN_MESSAGES: error
POSTGRESQL_SHARED_PRELOAD_LIBRARIES: pgaudit
Mounts:
/bitnami/postgresql from data (rw)
/dev/shm from dshm (rw)
/docker-entrypoint-initdb.d/ from custom-init-scripts (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-sessiondb-0
ReadOnly: false
custom-init-scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sessiondb-scheme
Optional: false
dshm:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned default/sessiondb-0 to ip-10-0-1-112.eu-central-1.compute.internal
Normal Created 11m (x4 over 11m) kubelet Created container sessiondb
Normal Started 11m (x4 over 11m) kubelet Started container sessiondb
Normal Pulled 10m (x5 over 11m) kubelet Container image "docker.io/bitnami/postgresql:11.13.0-debian-10-r33" already present on machine
Warning BackOff 110s (x57 over 11m) kubelet Back-off restarting failed container
---
% kubectl describe pvc data-sessiondb-0
Name: data-sessiondb-0
Namespace: default
StorageClass: standard
Status: Bound
Volume: pvc-6b56b20c-3e56-4a92-9278-794bf6cda4de
Labels: app.kubernetes.io/instance=asdf
app.kubernetes.io/name=sessiondb
role=primary
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 8Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: sessiondb-0
Events: <none>
---
% kubectl describe pvc data-sessiondb-0
Name: data-sessiondb-0
Namespace: default
StorageClass: standard
Status: Bound
Volume: pvc-6b56b20c-3e56-4a92-9278-794bf6cda4de
Labels: app.kubernetes.io/instance=asdf
app.kubernetes.io/name=sessiondb
role=primary
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 8Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: sessiondb-0
Events: <none>
cyrill@CyrillsMBP4380 core % kubectl describe pv pvc-6b56b20c-3e56-4a92-9278-794bf6cda4de
Name: pvc-6b56b20c-3e56-4a92-9278-794bf6cda4de
Labels: <none>
Annotations: hostPathProvisionerIdentity: 10bfa079-1086-4a77-849e-7d00de8e34dc
pv.kubernetes.io/provisioned-by: k8s.io/minikube-hostpath
Finalizers: [kubernetes.io/pv-protection]
StorageClass: standard
Status: Bound
Claim: default/data-sessiondb-0
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 8Gi
Node Affinity: <none>
Message:
Source:
Type: HostPath (bare host directory volume)
Path: /tmp/hostpath-provisioner/default/data-sessiondb-0
HostPathType:
Events: <none>
EDIT: Add logs with DEBUG level
--> Looking to fix: directory "/bitnami/postgresql/data" is not a database cluster directory
EDIT2: Add logs of the init container volumePermissions.enabled and permissions
EDIT3: Ok, so I created a fresh version to compare the old and the new one. I am wondering, why there is such a difference in the files as both were working until one didn't anymore. (During regular business, no upgrades, nothing.)
Upvotes: 2
Views: 4700
Answers (1)
Miaucl
Reputation: 364
I really hope nobody else runs across this, but finally I found the problem and for once it was not only between the chair and the monitor, but also RTFM was involved.
As mentioned I am using minikube to run my k8s cluster which provides PVs stored on the host disk. Where it is stored you may ask? Exaclty, here: /tmp/hostpath-provisioner/default/data-sessiondb-0/data/. You find the problem? No, I also took some time to figure it out. WHY ON EARTH does minikube use the tmp folder to store persistant volume claims?
This folder gets autom. cleared every now and so on.
SOLUTION: Change the path and DO NOT STORE PVs IN
tmp FOLDERS.
They mention this here: https://minikube.sigs.k8s.io/docs/handbook/persistent_volumes/#a-note-on-mounts-persistence-and-minikube-hosts and give an example.
But why use the "dangerous" tmp path per default and not, let's say, data without putting a Warning banner there?
Sigh. Closing this question ^^
--> Workaround: https://github.com/kubernetes/minikube/issues/7511#issuecomment-612099413
Github issues to this topic:
- https://github.com/kubernetes/minikube/issues/7511
- https://github.com/kubernetes/minikube/issues/13038
- https://github.com/kubernetes/minikube/issues/3318
- https://github.com/kubernetes/minikube/issues/5144
My Github issue for clarification in the docs: https://github.com/kubernetes/minikube/issues/13038#issuecomment-981821696
Upvotes: 5
Related Questions
- How to change bitnami postgresql helm chart configs? (ph_hba.conf, postgresql.conf)
- CrashLoopBackOff (postgres) - GCP
- How can I resolve kubernetes CrashLoopBackOff problem?
- PostgreSQL bitnami Helm Chart does not update the user password
- Keycloak stops without any reason using Bitnami Helm chart
- Bitnami Redis Cluster Helm Chart unable to bind to persistent volumes
- Postgres / K8S : PANIC could not locate a valid checkpoint record / CrashLoopBackOff
- Why am I getting a CrashLoopBackOff on this kubernetes example?
- Kubernetes Helm Chart - Debugging
- CrashLoopBackOff on kubernetes-dashboard