Unable to connect to mysql pod from app pod with in the same cluster and same namespace

Question

I am using minikube to deploy mysql and spring bootapp in the same namespace. For some reason I am unable to connect to the database from the app pod. Posting it here as I don't know where to go from here. Appreciate any help.

Can connect from my Laptop

I am able to connect to same mysql on minikube from my laptop both through MySQL workbench and through mysql command. I can run the app on my machine and the app is able to apply liquibase migrations to the database in the pod.

App cannot connect

When I try to apply deployment.yaml file, spring throws a huge stacktrace but these two lines summarise it all I think.

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

Caused by: java.net.ConnectException: Connection refused

Cannot connect even through MySQL client installed on the app pod

I installed MySQL client in the app pod using apt-get install -y mysql-client and then tried to connect to the MySQL db using the following commmands. I've also tried capture the dns log kubedns using kubectl logs -n kube-system -l k8s-app=kube-dns. The output of that command is attached below each mysql command.

Service name

mysql -u root -h syms-mysql -p
Enter password:
ERROR 2003 (HY000): Can't connect to MySQL server on 'syms-mysql:3306' (111)

log

[INFO] 172.17.0.1:60559 - 27285 "AAAA IN syms-mysql.syms.svc.cluster.local.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000323333s
[INFO] 172.17.0.1:60559 - 31377 "A IN syms-mysql.syms.svc.cluster.local.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000450412s
[INFO] 172.17.0.1:1440 - 42659 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000298098s
[INFO] 172.17.0.1:1440 - 17839 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000576619s
[INFO] 172.17.0.1:65385 - 46870 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000476572s
[INFO] 172.17.0.1:65385 - 39937 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000647759s
[INFO] 172.17.0.1:3786 - 45586 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000515706s
[INFO] 172.17.0.1:3786 - 60960 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.00063402s
[INFO] 172.17.0.1:7212 - 14821 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000255618s
[INFO] 172.17.0.1:7212 - 53452 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000563956s

DNS name

bash-4.4# mysql -u root -h syms-mysql.syms.svc.cluster.local -p
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on 'syms-mysql.syms.svc.cluster.local:3306' (111)

log

[INFO] 172.17.0.1:7212 - 14821 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000255618s
[INFO] 172.17.0.1:7212 - 53452 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000563956s
[INFO] 172.17.0.1:45103 - 8087 "AAAA IN syms-mysql.syms.svc.cluster.local.syms.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 167 0.001098626s
[INFO] 172.17.0.1:45103 - 25752 "A IN syms-mysql.syms.svc.cluster.local.syms.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 167 0.000285655s
[INFO] 172.17.0.1:14956 - 9219 "AAAA IN syms-mysql.syms.svc.cluster.local.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 162 0.000281106s
[INFO] 172.17.0.1:14956 - 1818 "A IN syms-mysql.syms.svc.cluster.local.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 162 0.000927891s
[INFO] 172.17.0.1:14683 - 58630 "AAAA IN syms-mysql.syms.svc.cluster.local.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000198551s
[INFO] 172.17.0.1:14683 - 7423 "A IN syms-mysql.syms.svc.cluster.local.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000984013s
[INFO] 172.17.0.1:2028 - 8676 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000984814s
[INFO] 172.17.0.1:2028 - 20460 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.001115167s

Cluster IP

bash-4.4# mysql -u root -h 10.110.201.105 -p
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on '10.110.201.105:3306' (111)

log

[INFO] 172.17.0.1:7212 - 14821 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000255618s
[INFO] 172.17.0.1:7212 - 53452 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000563956s
[INFO] 172.17.0.1:45103 - 8087 "AAAA IN syms-mysql.syms.svc.cluster.local.syms.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 167 0.001098626s
[INFO] 172.17.0.1:45103 - 25752 "A IN syms-mysql.syms.svc.cluster.local.syms.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 167 0.000285655s
[INFO] 172.17.0.1:14956 - 9219 "AAAA IN syms-mysql.syms.svc.cluster.local.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 162 0.000281106s
[INFO] 172.17.0.1:14956 - 1818 "A IN syms-mysql.syms.svc.cluster.local.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 162 0.000927891s
[INFO] 172.17.0.1:14683 - 58630 "AAAA IN syms-mysql.syms.svc.cluster.local.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000198551s
[INFO] 172.17.0.1:14683 - 7423 "A IN syms-mysql.syms.svc.cluster.local.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000984013s
[INFO] 172.17.0.1:2028 - 8676 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000984814s
[INFO] 172.17.0.1:2028 - 20460 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.001115167s

MySQL Manifests

Service

apiVersion: v1
kind: Service
metadata:
  name: syms-mysql
spec:
  ports:
  - protocol: TCP
    port: 3306
    targetPort: 3306
  selector:
    app: syms-mysql-db

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: syms-mysql
  labels:
    app: syms-mysql-db
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:8.0.31
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: syms-mysql-secret
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: syms-mysql-pvc

Persistent Volume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: syms-mysql-pv
  namespace: syms
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

Persistent Volume Claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: syms-mysql-pvc
  namespace: syms
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

App Manifests

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: syms-api
  namespace: syms
spec:
  selector:
    matchLabels:
      app: syms-api
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: syms-api
    spec:
      containers:
      - image: syms-api:0.0.1-SNAPSHOT
        name: syms-api
        env:
        - name: SPRING_DATASOURCE_PASSWORD
          value: test1234
        - name: SPRING_DATASOURCE_URL
          value: jdbc:mysql://syms-mysql:3306/syms?createDatabaseIfNotExist=true&useSSL=false&allowPublicKeyRetrieval=true
        - name: SPRING_DATASOURCE_USERNAME
          value: root
        ports:
        - containerPort: 8090
          name: syms-api

Service

apiVersion: v1
kind: Service
metadata:
  name: syms-api
  namespace: syms
spec:
  ports:
  - port: 8090
  selector:
    app: syms-api

Edit 1

Attached below is the app docker file I am using.

FROM adoptopenjdk/openjdk16

WORKDIR /code

ENV PORT 80

RUN java -version

RUN apt-get update && \
    apt-get install -y mysql-client

ENTRYPOINT ["tail", "-f", "/dev/null"]

Network Policies

I wasn't sure if I needed to create any network policies, but this is what the documentation says.

By default, if no policies exist in a namespace, then all ingress and egress traffic is allowed to and from pods in that namespace. The following examples let you change the default behavior in that namespace.

https://kubernetes.io/docs/concepts/services-networking/network-policies/#default-policies

Answer 1

Please update mysql deployment's selector "app: mysql" and template labels to match the one specified in the service.