CODEWITHSUNDEEP
hazelcast
Hello
Hello

Reputation: 1

Hazelcast 5.4.0 members couldn't connect to each other when Istio envoy proxy enabled

Issue: We recently upgraded hazelcast version from 3.12.13 to 5.4.0. Initially started with single pod.This single pod join the cluster(master).When we increased pod to 2 or more, Members couldn't establish connection to each other when deployed in Kubernetes cluster with Istio Envoy Proxy enabled

Versions:

The problem is that one pod goes on 2/2 without problems , the other one continue to crash and print below error

Log Snippet:
com.hazelcast.core.HazelcastException: java.util.concurrent.TimeoutException: JoinMastershipClaimOp failed to complete within 9999991909 NANOSECONDS. Invocation{op=com.hazelcast.internal.cluster.impl.operations.JoinMastershipClaimOp{serviceName='hz:core:clusterService', identityHash=1854415116, partitionId=-1, replicaIndex=0, callId=2, invocationTime=1656088715788 (2022-06-24 16:38:35.788), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=1, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1656088715788, firstInvocationTime='2022-06-24 16:38:35.788', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 00:00:00.000', target=[172.16.70.174]:5701, pendingResponse={VOID}, backupsAcksExpected=-1, backupsAcksReceived=0, connection=null}
    at com.hazelcast.internal.util.ExceptionUtil.lambda$static$0(ExceptionUtil.java:56)
    at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:134)
    at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:77)
    at com.hazelcast.internal.util.ExceptionUtil.rethrow(ExceptionUtil.java:139)
    at com.hazelcast.internal.util.FutureUtil$1.handleException(FutureUtil.java:53)
    at com.hazelcast.internal.util.FutureUtil.returnWithDeadline(FutureUtil.java:275)
    at com.hazelcast.internal.util.FutureUtil.returnWithDeadline(FutureUtil.java:240)
    at com.hazelcast.internal.cluster.impl.TcpIpJoiner.claimMastership(TcpIpJoiner.java:214)
    at com.hazelcast.internal.cluster.impl.TcpIpJoiner.tryClaimMastership(TcpIpJoiner.java:169)
    at com.hazelcast.internal.cluster.impl.TcpIpJoiner.joinViaPossibleMembers(TcpIpJoiner.java:155)
    at com.hazelcast.internal.cluster.impl.TcpIpJoiner.doJoin(TcpIpJoiner.java:96)
    at com.hazelcast.internal.cluster.impl.AbstractJoiner.join(AbstractJoiner.java:137)

Found below link related to the issue

But solution mentioned in the link ,that we have tried using appProtocol but that didn't work for us.

How can we prevent this ? how to make sure Hazelcast mark the connection properly between the pods ip and not the loopback?

Upvotes: 0

Views: 167

Answers (0)

Related Questions