Unable to connect to the postgres from spire server

Question

I have to set up a spire server with postgres database as backend. I am following this document - https://spiffe.io/docs/latest/try/getting-started-k8s.

After the setup the spire server and the spire agent pods were running. Then I changed the database in the server-configmap.yaml file from sqlite to postgres as below -

plugin_data {
    database_type = "postgres"
    connection_string = "dbname=postgres user=postgres password=Jj8Rt9tbyc host=my-postgresql port=5432"
}

Then I have started a postgres service in the same namespace as the spire server.

$ kubectl get svc my-postgresql -n spire
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
my-postgresql   ClusterIP   10.96.127.179   <none>        5432/TCP   4d2h

rsatal@rsatal-ub20:~/rahul$ kubectl get endpoints my-postgresql -n spire
NAME            ENDPOINTS           AGE
my-postgresql   10.244.0.201:5432   4d2h

But on the spire server logs, I am getting this error -

$ k logs spire-server-0 
time="2024-09-13T10:00:42Z" level=warning msg="Current umask 0022 is too permissive; setting umask 0027"
time="2024-09-13T10:00:42Z" level=info msg=Configured admin_ids="[]" data_dir=/run/spire/data
time="2024-09-13T10:00:42Z" level=info msg="Opening SQL database" db_type=postgres subsystem_name=sql
time="2024-09-13T10:00:47Z" level=error msg="Fatal run error" error="datastore-sql: dial tcp: lookup my-postgresql: Try again"
time="2024-09-13T10:00:47Z" level=error msg="Server crashed" error="datastore-sql: dial tcp: lookup my-postgresql: Try again"

I have no idea what's going wrong and why the spire server pod is not able to access the postgres. I am also not able to get inside the spire server -

 kubectl exec -it spire-server-0 -- nslookup my-postgresql
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "0089c4fa4e670cc624f5e24d00b27b06356f936345f19c70cf2545e1cf18a78d": OCI runtime exec failed: exec failed: unable to start container process: exec: "nslookup": executable file not found in $PATH: unknown

But to debug it I have tried creating a debug container and I was able to access the postgres

$ kubectl run -it --rm debug2 --image=busybox --restart=Never -- sh
If you don't see a command prompt, try pressing enter.
/ # 
/ # nslookup my-postgresql
Server:         10.96.0.10
Address:        10.96.0.10:53

Answer 1

If it helps anyone I recently spun up such a demo deploymemt using the new Helm charts which enable you to ship tested and compatible resources. Use a Helm values file:

global:
  spire:
    clusterName: oauth
    trustDomain: oauthdemo.example

spire-server:
  replicaCount: 1
  caKeyType: ec-p256
  dataStore:
    sql:
      host: spire-database
      databaseType: postgres
      databaseName: spire
      username: spire
      password: Password1
      options:
      - sslmode: disable

Then run helm template to see how the authors manage Postgres related settings in the server configmap. In my case I just created a basic Postgres deployment with these settings.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spire-database
---
kind: Service
apiVersion: v1
metadata:
  name: spire-database
spec:
  ports:
  - protocol: TCP
    port: 5432
    targetPort: postgres
  selector:
    app: spire-database
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: spire-database
  labels:
    app: spire-database
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spire-database
  serviceName: spire-database
  template:
    metadata:
      namespace: spire
      labels:
        app: spire-database
    spec:
      containers:
        - name: spire-database
          image: $DOCKER_IMAGE
          volumeMounts:
          - name: postgres-data
            mountPath: /var/lib/postgresql/data
            readOnly: false
          env:
            - name: POSTGRES_DB
              value: spire
            - name: POSTGRES_USER
              value: spire
            - name: POSTGRES_PASSWORD
              value: Password1
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          ports:
            - name: postgres
              containerPort: 5432
          livenessProbe:
            tcpSocket:
              port: 5432
            failureThreshold: 2
            initialDelaySeconds: 30
            periodSeconds: 60
            timeoutSeconds: 3
      volumes:
        - name: postgres-data
          persistentVolumeClaim:
            claimName: spire-data-spire-server-0