CODEWITHSUNDEEP
spring-bootkuberneteshazelcast
ViV
ViV

Reputation: 2118

hazelcast-kubernetes Unable to start Embedded server

I'm trying to start a simple embedded hazelcast server in a spring boot application on kubernetes. I'm using hazelcast-kubernetes plugin and have followed the documentation here. I tried both LoadBalancer and NodePort service types.

I also tried to deploy the sample application using this example (using the same docker image) and tried both LoadBalancer and NodePort service types.

In both cases I get the below error after 10 retries to connect to Kubernetes Master. Where a.b.c.d is the IP address of the kubernetes master node.

2020-03-09 18:20:57.322  INFO 1 --- [           main] c.h.c.LifecycleService                   : [a.b.c.d]:5701 [dev] [3.12] [a.b.c.d]:5701 is STARTING
2020-03-09 18:20:57.820  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [1] retrying in 1 seconds...
2020-03-09 18:20:59.339  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [2] retrying in 2 seconds...
2020-03-09 18:21:01.713  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [3] retrying in 3 seconds...
2020-03-09 18:21:05.113  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [4] retrying in 5 seconds...
2020-03-09 18:21:10.198  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [5] retrying in 7 seconds...
2020-03-09 18:21:17.807  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [6] retrying in 11 seconds...
2020-03-09 18:21:49.216  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [7] retrying in 17 seconds...
2020-03-09 18:22:06.319  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [8] retrying in 25 seconds...
2020-03-09 18:22:31.963  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [9] retrying in 38 seconds...
2020-03-09 18:23:15.423  WARN 1 --- [           main] c.h.k.RetryUtils                         : Couldn't connect to the Kubernetes master, [10] retrying in 57 seconds...
2020-03-09 18:24:23.099 ERROR 1 --- [           main] c.h.i.c.i.DiscoveryJoiner                : [a.b.c.d]:5701 [dev] [3.12] Failure in KubernetesClient
com.hazelcast.kubernetes.KubernetesClientException: Failure in KubernetesClient
        at com.hazelcast.kubernetes.DefaultKubernetesClient.callGet(DefaultKubernetesClient.java:109) ~[hazelcast-kubernetes-1.3.1.jar!/:?]
        at com.hazelcast.kubernetes.DefaultKubernetesClient.endpointsByName(DefaultKubernetesClient.java:88) ~[hazelcast-kubernetes-1.3.1.jar!/:?]
        .
        .
        .       
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [main.jar:?]
Caused by: java.net.UnknownHostException: kubernetes.default.svc
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[?:1.8.0_111]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_111]

In the second case, where I'm using leszko's sample, the ip address it tries to connect to is the IP of the POD.

[190.168.1.194]:5701 [dev] [3.11] Failure in KubernetesClient

190.168.1.194 is the POD's IP address

I added the below env to the deployment yaml, but doesn't look like it is overriding this varaible.

env:
           - name: service-name
             value: "hazelcast-embedded"

Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: null, service-port: 0, service-label: null, service-label-value: true, namespace: default, resolve-not-ready-addresses: false, kubernetes-master: https://kubernetes.default.svc}

Upvotes: 1

Views: 2244

Answers (1)

Rafał Leszko
Rafał Leszko

Reputation: 5531

You seem to have an issue with your Kubernetes cluster itself.

  1. First you try to use Hazelcast Kubernetes Discovery strategy Kubernetes API, which makes a REST call to Kubernetes master. Kubernetes master should be always accessible from your POD by the hostname kubernetes.default.svc. In your case, it's not found.

  2. Then, you try to use Hazelcast Kubernetes Discovery strategy DNS Lookup, which uses service DNS to resolve Hazelcast members. In your case, as mentioned by @Mesut, it's not resolving correctly the DNS of a service.

Upvotes: 1

Related Questions