CODEWITHSUNDEEP
javaspringpostgresqlspring-bootkubernetes
joe1531
joe1531

Reputation: 514

Can't connect spring boot to postgres k8s statefulset

I am new to Kubernetes and I am deploying an application for the first time on Kubernetes. I want to deploy a postgreSQL statefulset and a simple replicaset of spring boot pods. I created a headless service that will be attached to the following statefulset.

    # Headless Service
apiVersion: v1
kind: Service
metadata:
  name: ecom-db-h
spec:
  ports:
  - targetPort: 5432
    port: 80
  clusterIP: None
  selector:
    app: ecom-db
---
# PostgreSQL StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ecomdb-statefulset
  labels:
    app: ecom-db
spec:
  serviceName: ecom-db-h
  selector:
    matchLabels:
      app: postgresql-db
  replicas: 2
  template:
    # Pod Definition
    metadata:
      labels:
        app: postgresql-db
    spec:
      containers:
        - name: db
          image: postgres:13
          ports:
            - name: postgres
              containerPort: 5432
         # volumeMounts:
          #  - name: ecom-db
           #   mountPath: /var/lib/postgresql/data
          env:
            - name: POSTGRES_DB
              value: ecommerce
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              value: <password>
      #volumes:
       # - name: ecom-db
        #  persistentVolumeClaim:
         #   claimName: ecom-pvc

And here is the replicaset I created for the spring boot application pods :

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: ecom-server
  labels:
    app: ecom-server
    tier: backend
spec:
  # modify replicas according to your case
  replicas: 2
  selector:
    matchLabels:
      type: backend
  template:
    # Pod definition
    metadata:
      labels:
        app: ecom-server
        type: backend
    spec:
      containers:
      - name: ecommerce-server
        image: <my.private.repo.url>/spring-server:kubernetes-test
        imagePullPolicy: Always
      imagePullSecrets:
          - name: regcred

I am using the default namespace and here is the application.properties:

spring.datasource.url=jdbc:postgresql://ecom-db-h.default.svc.cluster.local/ecommerce?useSSL=false
spring.datasource.driverClassName=org.postgresql.Driver
spring.datasource.username=postgres
spring.datasource.password=<password>
spring.jpa.hibernate.ddl-auto=update
spring.jpa.hibernate.format_sql=true
spring.jpa.hibernate.id.new_generator_mappings=true
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL95Dialect

But my spring boot containers always exit with the following error :

Caused by: java.net.UnknownHostException: ecom-db-h.default.svc.cluster.local
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.postgresql.core.PGStream.createSocket(PGStream.java:231)
        at org.postgresql.core.PGStream.<init>(PGStream.java:95)
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:98)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
        ... 136 common frames omitted

This means that either the replicaset is not able to find the headless service, or my headless service is not connected to the statefulset pods at all. Did I miss something in the YAML files ? Or am I using the headless service in the wrong way ?

Update:

Thanks to @Thomas and @harsh-manvar I understood that the headless service is not meant to connect to the databases. Instead, I should use a normal ClusterIP service. The problem is I did that and I am still getting the same error. Here is my new YAML files :

application.properties modification:

spring.datasource.url=jdbc:postgresql://db-svc.default.svc.cluster.local/ecommerce?useSSL=false

statefulset.yaml modified

# ClusterIP service instead of the headless service
apiVersion: v1
kind: Service
metadata:
  name: db-svc
spec:
  ports:
  - name: pgql
    port: 80
    targetPort: 5432
    protocol: TCP
  selector:
    app: db
---
# PostgreSQL StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db-sts
  labels:
    app: db-sts
spec:
  serviceName: db-svc
  selector:
    matchLabels:
      app: db
  replicas: 2
  template:
    # Pod Definition
    metadata:
      labels:
        app: db
    spec:
      containers:
        - name: db
          image: postgres:13
          ports:
            - name: postgres
              containerPort: 5432
         # volumeMounts:
          #  - name: ecom-db
           #   mountPath: /var/lib/postgresql/data
          env:
            - name: POSTGRES_DB
              value: ecommerce
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              value: mss#123
      #volumes:
       # - name: ecom-db
        #  persistentVolumeClaim:
         #   claimName: ecom-pvc

Here is the new error :

Caused by: java.net.UnknownHostException: db-svc.default.svc.cluster.local
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.postgresql.core.PGStream.createSocket(PGStream.java:231)
        at org.postgresql.core.PGStream.<init>(PGStream.java:95)
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:98)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
        ... 136 common frames omitted

My statefulset pods are running smoothly and the service is created :

NAME                      READY   AGE
statefulset.apps/db-sts   2/2     26m

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/db-svc       ClusterIP   10.108.39.189   <none>        5432/TCP   26m
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP    30m

NAME                READY   STATUS             RESTARTS        AGE
db-sts-0            1/1     Running            0               26m
db-sts-1            1/1     Running            0               26m
ecom-server-5jbs2   0/1     CrashLoopBackOff   9 (44s ago)     25m
ecom-server-rtpmg   0/1     CrashLoopBackOff   8 (4m46s ago)   25m

NAME         ENDPOINTS                           AGE
db-svc       10.244.1.48:5432,10.244.2.85:5432   18h

Second Update :

After running an alpine-shell container in a pod to check on the DNS resolving, I performed an "nslookup db-svc.default.svc.cluster.local" and I got this :

nslookup db-svc.default.svc.cluster.local
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   db-svc.default.svc.cluster.local
Address: 10.107.101.52

which means the service does exist and it can be reached from this container. What's strange is when I performed the pgsql command to try to connect to the postgres database server, I got no response at all and the command did not even exit. Here is the error

WARN  SqlExceptionHelper - SQL Error: 0, SQLState: 08001
ERROR SqlExceptionHelper - The connection attempt failed.
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:315)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)
        at org.postgresql.Driver.makeConnection(Driver.java:465)
        at org.postgresql.Driver.connect(Driver.java:264)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358)
        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477)
        at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560)
        at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
        at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
        at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator$ConnectionProviderJdbcConnectionAccess.obtainConnection(JdbcEnvironmentInitiator.java:180)
        at org.hibernate.resource.transaction.backend.jdbc.internal.DdlTransactionIsolatorNonJtaImpl.getIsolatedConnection(DdlTransactionIsolatorNonJtaImpl.java:43)
        ... 122 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at org.postgresql.core.PGStream.createSocket(PGStream.java:231)
        at org.postgresql.core.PGStream.<init>(PGStream.java:95)
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:98)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
        ... 136 common frames omitted

Upvotes: 1

Views: 2040

Answers (3)

joe1531
joe1531

Reputation: 514

Finally I found the solution :

Original Post: First I need to use the ClusterIP service and not the headless service.

Solution to Update : It was the firewall blocking the requests between the pods inside my serves. That is why I was getting the error SQLState: 08001. Disabling the firewall solved the problem. Also I needed to specify the port in the application.properties. Because I got used to not specify the port 80 with HTTP connections, I thought that was the case for the postgres connection to the database, but I needed to specify the port :

spring.datasource.url=jdbc:postgresql://db-svc.default.svc.cluster.local:80/ecommerce?useSSL=false

It would be a better practise to change the service port to the default postgres server port: 5432

Upvotes: 0

Harsh Manvar
Harsh Manvar

Reputation: 30103

Headless service won't give you DNS record you should be any of ClusterIP or NodePort or LoadBalancer

A headless service is a service with a service IP but instead of load-balancing it will return the IPs of our associated Pods. This allows us to interact directly with the Pods instead of a proxy. It's as simple as specifying None for .spec.clusterIP and can be utilized with or without selectors - you'll see an example with selectors in a moment.

Read more about headless serivce at : https://dev.to/kaoskater08/building-a-headless-service-in-kubernetes-3bk8

you can follow below YAML file which is creating the serivce type : ClusterIP

apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  ports:
  - name: pgql
    port: 5432
    targetPort: 5432
    protocol: TCP
  selector:
    app: postgres
---
apiVersion: apps/v1  
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: "postgres"
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:9.5
        volumeMounts:
        - name: postgres-data
          mountPath: /var/lib/postgresql/data
          subPath: pgdata
        env:
        - name: POSTGRES_USER
          value: root
        - name: POSTGRES_PASSWORD
          value: password
        - name: POSTGRES_DB
          value: kong
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        ports:
        - containerPort: 5432
      terminationGracePeriodSeconds: 60
  volumeClaimTemplates:
  - metadata:
      name: postgres-data
    spec:
      accessModes:
      - "ReadWriteOnce"
      resources:
        requests:
          storage: 10Gi

https://github.com/harsh4870/Keycloack-postgres-kubernetes-deployment/blob/main/postgres.yaml

Upvotes: 1

Thomas
Thomas

Reputation: 12009

The definition of a headless service is to not provide a DNS record and provide internal load balancing.

A headless service can be used to query the endpoints and handle them separately.

To fix your issue create a regular service.

Upvotes: 1

Related Questions