CODEWITHSUNDEEP
mysqlkuberneteskubernetes-helmreadinessprobelivenessprobe
Navin Kumar
Navin Kumar

Reputation: 41

Mysql Kubernetes Deployment helm chart fails with readiness and liveness probe failed

I am new to stackoverflow, so pardon if I have not followed all rules for asking this question.

So, I am using this helm chart: https://github.com/helm/charts/tree/master/stable/mysql for deploying mysql for our production env. Infact we have this setup in our production running but it creates only 1 pod of mysql whereas I need to run 3 pods. I tried to set the replicas=3 in values.yaml file of the helm chart and redeploy that chart but it is continuously failing with some error, the error being:

Events:
    Type     Reason            Age                    From                  Message
    ----     ------            ----                   ----                  -------
    Warning  FailedScheduling  4m58s                  default-scheduler     pod has unbound immediate PersistentVolumeClaims (repeated 4 times)
    Normal   Scheduled         4m58s                  default-scheduler     Successfully assigned test-helm/cluster-mysql-5bcdf87779-vdb2s to innolx12896
    Normal   Pulled            4m57s                  kubelet, innolx12896  Container image "busybox:1.29.3" already present on machine
    Normal   Created           4m56s                  kubelet, innolx12896  Created container
    Normal   Started           4m56s                  kubelet, innolx12896  Started container
    Warning  BackOff           4m32s (x2 over 4m37s)  kubelet, innolx12896  Back-off restarting failed container
    Normal   Pulled            2m52s (x4 over 4m55s)  kubelet, innolx12896  Container image "mysql:5.7.14" already present on machine
    Normal   Created           2m52s (x4 over 4m55s)  kubelet, innolx12896  Created container
    Warning  Unhealthy         2m52s (x3 over 3m12s)  kubelet, innolx12896  Readiness probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
                                                                            mysqladmin: connect to server at 'localhost' failed
                                                                            error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
                                                                            Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
    Normal   Killing           2m52s                  kubelet, innolx12896  Killing container with id docker://cluster-mysql:Container failed liveness probe.. Container will be killed and recreated.
    Normal   Started           2m51s (x4 over 4m54s)  kubelet, innolx12896  Started container
    Warning  Unhealthy         2m12s (x4 over 3m42s)  kubelet, innolx12896  Liveness probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
                                                                            mysqladmin: connect to server at 'localhost' failed
                                                                            error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
                                                                            Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!

Also when I try to see the mysqld sock, it is missing. But I don't understand why. Since a single pod deployment works fine but 3 pod deployment produces this error.

Also there might be ways to solve this problem by explicitly defining the paths for the sock which other StackOverflow answers suggest like this answer 1 or this answer 2. But the caveat is we have whole production chart deployment automated through Jenkins and there is no scope of explicitly running those commands for every production release.

So, I need to know how to solve this problem or maybe know what is causing this problem, so that I can change or alter some parameters in the helm chart I mentioned in the beginning. Also tell me if it is not possible to scale this deployment, meaning this helm chart doesn't support or is not meant for multiple mysql pods.

Thanks in advance.

Upvotes: 4

Views: 6152

Answers (1)

itsmeremz
itsmeremz

Reputation: 61

It looks like the Readiness probe failed & the Liveness probe is failing in your case. For MySQL, please try with the below (in templates/deployment.yaml) for Readiness & Liveness:

ports:
  - name: mysql
    containerPort: {{ .Values.service.port }}
    protocol: TCP
livenessProbe:
  exec:
    command: ["mysqladmin", "ping"]
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
readinessProbe:
  exec:
    # Check we can execute queries over TCP (skip-networking is off).
    command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
  initialDelaySeconds: 5
  periodSeconds: 2
  timeoutSeconds: 1

Upvotes: 6

Related Questions