Reputation: 198

Why does the pods hostname don't resolve?

i have a problem with pods in kubernetes. i have an app pod (invoice) with init container that checks if mysql pod (invoice-mysql) is running invoice-mysql is running and ready but the init container in the invoice pod is not seeing it

here is the logs of the init container

DB is not yet reachable;sleep for 10s before retry
DB is not yet reachable;sleep for 10s before retry
nc: bad address 'invoice-mysql'
nc: bad address 'invoice-mysql'
DB is not yet reachable;sleep for 10s before retry
nc: bad address 'invoice-mysql'
DB is not yet reachable;sleep for 10s before retry
nc: bad address 'invoice-mysql'

here is the yaml of invoice

apiVersion: apps/v1
kind: Deployment
metadata:
  name: invoice
  namespace: jhipster
spec:
  replicas: 1
  selector:
    matchLabels:
      app: invoice
      version: 'v1'
  template:
    metadata:
      labels:
        app: invoice
        version: 'v1'
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - invoice
                topologyKey: kubernetes.io/hostname
              weight: 100
      initContainers:
        - name: init-ds
          image: busybox:latest
          command:
            - '/bin/sh'
            - '-c'
            - |
              while true
              do
                rt=$(nc -z -w 1 invoice-mysql 3306)
                if [ $? -eq 0 ]; then
                  echo "DB is UP"
                  break
                fi
                echo "DB is not yet reachable;sleep for 10s before retry"
                sleep 10
              done
      containers:
        - name: invoice-app
          image: docker.pkg.github.com/morsi84/kubernetes/invoice
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: prod
            - name: SPRING_CLOUD_CONFIG_URI
              value: http://admin:${jhipster.registry.password}@jhipster-registry.Jhipster.svc.cluster.local:8761/config
            - name: JHIPSTER_REGISTRY_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: registry-secret
                  key: registry-admin-password
            - name: EUREKA_CLIENT_SERVICE_URL_DEFAULTZONE
              value: http://admin:${jhipster.registry.password}@jhipster-registry.Jhipster.svc.cluster.local:8761/eureka/
            - name: SPRING_DATASOURCE_URL
              value: jdbc:mysql://invoice-mysql.Jhipster.svc.cluster.local:3306/invoice?useUnicode=true&characterEncoding=utf8&useSSL=false&useLegacyDatetimeCode=false&serverTimezone=UTC&createDatabaseIfNotExist=true
            - name: SPRING_SLEUTH_PROPAGATION_KEYS
              value: 'x-request-id,x-ot-span-context'
            - name: JAVA_OPTS
              value: ' -Xmx256m -Xms256m'
          resources:
            requests:
              memory: '512Mi'
              cpu: '500m'
            limits:
              memory: '1Gi'
              cpu: '1'
          ports:
            - name: http
              containerPort: 8081
          readinessProbe:
            httpGet:
              path: /management/health
              port: http
            initialDelaySeconds: 20
            periodSeconds: 15
            failureThreshold: 6
          livenessProbe:
            httpGet:
              path: /management/health
              port: http
            initialDelaySeconds: 120
      imagePullSecrets:
          - name: regcred

here is the yaml of invoice-mysql

apiVersion: apps/v1
kind: Deployment
metadata:
  name: invoice-mysql
  namespace: jhipster
spec:
  replicas: 1
  selector:
    matchLabels:
      app: invoice-mysql
  template:
    metadata:
      labels:
        app: invoice-mysql
    spec:
      volumes:
        - name: data
          emptyDir: {}
      containers:
        - name: mysql
          image: mysql:8.0.20
          env:
            - name: MYSQL_USER
              value: root
            - name: MYSQL_ALLOW_EMPTY_PASSWORD
              value: 'yes'
            - name: MYSQL_DATABASE
              value: invoice
          args:
            - --lower_case_table_names=1
            - --skip-ssl
            - --character_set_server=utf8mb4
            - --explicit_defaults_for_timestamp
          ports:
            - containerPort: 3306
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql/
          resources:
            requests:
              memory: '512Mi'
              cpu: '500m'
            limits:
              memory: '1Gi'
              cpu: '1'
---
apiVersion: v1
kind: Service
metadata:
  name: invoice-mysql
  namespace: jhipster
spec:
  selector:
    app: invoice-mysql
  ports:
    - port: 3306

here is a description of the invoice-mysql service

Name:              invoice-mysql
Namespace:         jhipster
Labels:            <none>
Annotations:       <none>
Selector:          app=invoice-mysql
Type:              ClusterIP
IP:                10.98.220.110
Port:              <unset>  3306/TCP
TargetPort:        3306/TCP
Endpoints:         10.244.1.66:3306
Session Affinity:  None
Events:            <none>

working environment: