CODEWITHSUNDEEP
spring-bootopenshifthazelcastleader-election
Paul M
Paul M

Reputation: 3

Can Hazelcast Leader Election Clustering work in OpenShift Containers without hz operators?

I am trying to deploy a Java SpringBoot microservice to make use of Leader Election on OpenShift Container. I was able to get it to work locally with no configuration. I tried 2 nodes (on the same IP, different ports). I did no special configuration for it to work.

But when I deployed my code to OpenShift Container, and deployed it to 2 nodes, the clustering is not working. The nodes can't seem to find each other (they are on different IPs, same port). So it is split-brained.

I looked up how to do this, and found these instructions to make use of and install hazelcast-operators (Hazelcast platform operators) to get it to work, but in my particular environment, we are not allowed to install such packages.

Is there any way for me to this working, without needing to install hazelcast operators, RBAC rules, or without needing to do anything to the OpenShift environment setup (I work in a very restricted OpenShift environment)?

Is there any programmatic or YAML configuration (at the application level) that I can deploy from my code to get this working?

I tried deploying my Microservice. This is essentially the code I have:

@Component
public class Runner implements CommandLineRunner, MembershipListener {
    private static final Logger logger = Logger.getLogger(Runner.class.getName());
...
    private String leaderAddress;
    private HazelcastInstance hazelcastInstance;
...    
    @Autowired
    private CSPRabbitMQConfig rabbitMQConfiguration;

    public Runner(RabbitTemplate rabbitTemplate) {
        this.rabbitTemplate = rabbitTemplate;
    }
    @Override
    public void run(String... args) throws Exception {      
        hazelcastInstance = Hazelcast.newHazelcastInstance(/* hazelcastConfig */);
        hazelcastInstance.getCluster().addMembershipListener(this);
        
        logger.info("Members: "+ hazelcastInstance.getCluster().getMembers().size());
        leaderAddress = hazelcastInstance.getCluster().getMembers().iterator().next().getSocketAddress().toString();
        logger.info("Current leader address: " + leaderAddress);
...
    // Leader should be the only process to start the subscriptions
    if (amITheLeader()) {
        // TODO: need to verify this works in a multi-host environment.
        // This node is the leader, do leader-specific tasks here
...
        }
...
    }
    ...
    public boolean amITheLeader() {
        logger.info("> amITheLeader() - cluster size: " + hazelcastInstance.getCluster().getMembers().size());
        return hazelcastInstance.getCluster().getLocalMember().getSocketAddress().toString().equals(leaderAddress);
    }

    @Override
    public void memberAdded(MembershipEvent membershipEvent) {
        if (membershipEvent.getMember().getSocketAddress().toString().compareTo(leaderAddress) < 0) {
            // A new member has joined that has an older address, it becomes the new leader
            leaderAddress = membershipEvent.getMember().getSocketAddress().toString();
            if (amITheLeader()) {
                // This node is the new leader, do leader-specific tasks here
            }
        }
    }

    @Override
    public void memberRemoved(MembershipEvent membershipEvent) {
        if (membershipEvent.getMember().getSocketAddress().toString().equals(leaderAddress)) {
            // The leader has left the cluster, a new leader will be elected
            leaderAddress = hazelcastInstance.getCluster().getMembers().iterator().next().getSocketAddress().toString();
            if (amITheLeader()) {
                // This node is the new leader, do leader-specific tasks here
            }                  
        }
    }

And I included this maven dependency:

        <dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast-spring</artifactId>
        </dependency>

Perhaps there is another way to code this, or maybe I am doing this in an out of date way?

This works fine locally with multiple instances of the application running on same IP address (different ports) - sometimes, at other times, the nodes can't find one another.

I was expecting this to work automatically once deployed to OpenShift, but once I did, I saw it the nodes were not finding each other, and the cluster therefore was not working, which doesn't allow leader election to work properly.

Upvotes: 0

Views: 134

Answers (1)

S&#246;ren Henning
S&#246;ren Henning

Reputation: 396

Using Kubernetes in DNS Lookup Mode could be what you need.

The basic idea is that you create a Headless Service for your microservice in your Kubernetes cluster. It is basically an DNS entry pointing to all your service instances. Then, you configure this DNS name in your Hazelcast join configuration and each instance will find all the other instances. The documentation provides details on how such a Kubernetes Service could look like and how to adjust the Hazelcast configuration, either using YAML or Java.

The documentation provides also a list of pros and cons of this approach. My impression is that this DNS Lookup Mode is less used in production, but it seems to be still supported.

Upvotes: 0

Related Questions