CODEWITHSUNDEEP
javaredistestcontainersredis-cluster
djGowda
djGowda

Reputation: 49

How to bring up redis cluster using testContainers in java

I'm trying to create a Redis cluster using TestContainers to test my application, which depends on a Redis cluster. Here's what I have tried so far:

Code snippet for starting container:

Network network = Network.newNetwork();
RedisContainer redisContainer = new RedisContainer(DockerImageName.parse("redis:7.0.5"))
                    .withExposedPorts(port)
                    .withCommand("redis-server --port " + port +
                            " --requirepass " + redisPassword +  // Password for clients
                            " --masterauth " + redisPassword +  // Password for inter-node communication
                            " --cluster-enabled yes" +
                            " --cluster-config-file nodes.conf"+
                            " --cluster-node-timeout 5000"+
                            " --appendonly yes" +
                            " --bind 0.0.0.0" )
                    .withNetwork(network)
                    .withNetworkMode("bridge")
                    .withNetworkAliases("redis-" + i)
                    .waitingFor(Wait.forListeningPort());

1. Single-Node Cluster

I attempted to create a single-node cluster with cluster-enabled set to yes and replica set to 0. I tried connecting to it using JedisCluster.

Issues and Fixes:

Initially, I got the error: Cluster slots not allocated. I resolved this by running the CLUSTER ADDSLOTS command to allocate the slot range. After this, I ran the CLUSTER NODES command and got the following output:

1f2673c5fdb45ca16d564658ff88f815db5cbf01 172.29.0.2:6379@16379 myself,master - 0 0 1 connected 0-16383

However, when I tried connecting to the cluster using JedisCluster, connection got established, I was able to get nodes list with ip and port using jedisCluster.getClusterNodes() api. When I tried writing some key value pair got below error after few seconds.

redis.clients.jedis.exceptions.JedisClusterOperationException: Cluster retry deadline exceeded.

Oddly enough, running commands via redis-cli worked perfectly for both writing and reading data. Cluster Info Output:

cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:1
cluster_current_epoch:1
cluster_my_epoch:1
cluster_stats_messages_pong_sent:1
cluster_stats_messages_meet_sent:1
cluster_stats_messages_sent:2
cluster_stats_messages_pong_received:1
cluster_stats_messages_meet_received:1
cluster_stats_messages_received:2
total_cluster_links_buffer_limit_exceeded:0

Cluster Slots Output:

1) 1) (integer) 0
   2) (integer) 16383
   3) 1) "172.29.0.2"
      2) (integer) 6379
      3) "b47f7da9be31ce953d4b4fbf9e3a737d1c9b7a58"
      4) (empty array)

2. Multi-Node Cluster

I also tried setting up a 6-node cluster (3 masters and 3 slaves).

Observations:

JedisCluster.getClusterNodes() returned the correct node information for all 3 master nodes (IP and port). However, when I tried writing data to the cluster using JedisCluster, I got the following error:

redis.clients.jedis.exceptions.JedisClusterOperationException: Cluster retry deadline exceeded.

When I used redis-cli -c to write data, it got stuck at the Redirecting to slot [<some slot>] located at <ip> node message.

Possible Issue I'm trying to bring up redis in a container using TestContainer module. I suspect the nodes in the cluster are unable to communicate with each other properly. In the case of the single-node cluster, some configuration might still be missing. Some

Any help in resolving this issue would be greatly appreciated. Thanks!

Edit:

JedisCluster Config code:

// Using redisContainer in above code snippet start the container and run  <redis-cli --no-auth-warning -h localhost -p 6379 -a password cluster addslotsrange 0 16383> to add slots and continue with below code

        GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
        poolConfig.setMaxTotal(Runtime.getRuntime().availableProcessors()); 
        poolConfig.setMaxIdle(Runtime.getRuntime().availableProcessors());
        poolConfig.setMinIdle(Runtime.getRuntime().availableProcessors());
        poolConfig.setMaxWaitMillis(2000);

//        Connect to the cluster using Jedis with a password
        DefaultJedisClientConfig.Builder jedisClientConfig = DefaultJedisClientConfig.builder()
                .password(redisPassword)
                .ssl(false)
                .connectionTimeoutMillis(10000)
                .socketTimeoutMillis(4000);

        final Set<HostAndPort> hosts = new HashSet<>(redisContainers.size());
        int i = 0;
        for(RedisContainer redisContainer : redisContainers){
            String redisHost = "127.0.0.1";
            int redisPort = redisContainer.getMappedPort(basePort+i);
            hosts.add(new HostAndPort(redisHost, redisPort));
            i += 1;
        }

        System.out.println("Hosts " + hosts);

        try (JedisCluster jedis = new JedisCluster(hosts, jedisClientConfig.build(), 3, poolConfig)) {
            Map<String, redis.clients.jedis.ConnectionPool> nodes = jedis.getClusterNodes();
            System.out.println("Connected cluster nodes: " + nodes);
            nodes.forEach((key, value) -> System.out.println(key));
            jedis.set("key", "value"); // This is where the error is seen
            System.out.println("Key set in Redis Cluster: " + jedis.get("key"));
        }
    ```

Upvotes: 0

Views: 83

Answers (1)

Ivo Gaydazhiev
Ivo Gaydazhiev

Reputation: 26

In regards to "Single-Node Cluster":

The problem arises because .withExposedPorts(port) exposes the Redis service on a dynamically allocated local port. Meanwhile, the JedisCluster client uses the seed nodes (provided hosts) to resolve the cluster topology via the CLUSTER SLOTS or CLUSTER NODES command. Then, it will use host/port announced by the nodes themself to create connections to a particular node. As you can see from the output you have provided cluster nodes will announce the actual port they are running on (6379) unless cluster-announce-port is specified.

1f2673c5fdb45ca16d564658ff88f815db5cbf01 172.29.0.2:6379@16379 myself,master ...

Since port 6379 is not accessible outside the docker container (e.g., the test container exposes it on a different dynamically mapped port), call to jedis.set("key", "value"); will try to acquire connection to the node using the announced host/port and will fail.

You can overcome this by using statically mapped port bindin or use Jedis provided option for host/port mapping -DefaultJedisClientConfig.Builder#hostAndPortMapper.

Option 1: Expose redis service on predefined port

int externalPort = 7379;
int port = 6379;
Network network = Network.newNetwork();
RedisContainer redisContainer = new RedisContainer(DockerImageName.parse("redis:7.0.5"))
    // Use static port binding together with  cluster-announce-port
    .withCreateContainerCmdModifier(cmd -> cmd.withPortBindings(
        new PortBinding(Ports.Binding.bindPort(externalPort), ExposedPort.tcp(port))))
    .withCommand("redis-server --port " + port +
        " --requirepass " + redisPassword +  // Password for clients
        " --masterauth " + redisPassword +  // Password for inter-node communication
        " --cluster-announce-port " + externalPort +
        " --cluster-enabled yes" +
        " --cluster-config-file nodes.conf"+
        " --cluster-node-timeout 5000"+
        " --appendonly yes" +
        " --bind 0.0.0.0" )
    .withNetwork(network)
    .withNetworkMode("bridge")
    .withNetworkAliases("redis-" + i)
    .waitingFor(Wait.forListeningPort());

Option 2 : Use Jedis hostAndPortMapper

HostAndPortMapper nat = hostAndPort -> {
  if (hostAndPort.getPort() == port) {
    return new HostAndPort(redisContainer.getHost(), redisContainer.getMappedPort(port));
  }
  return hostAndPort;
};
...
//        Connect to the cluster using Jedis with a password
DefaultJedisClientConfig.Builder jedisClientConfig = DefaultJedisClientConfig.builder()
    .password(redisPassword)
    .hostAndPortMapper(nat)
    .ssl(false)
    .connectionTimeoutMillis(10000)
    .socketTimeoutMillis(4000);

Also, make sure the cluster has reached a stable state after slots were configured.

Here is a more complete example using Option 2 with hostAndPortMapper. This is rough code and is provided purely as an example for demonstration purposes:

import com.github.dockerjava.api.model.*;
import com.redis.testcontainers.RedisContainer;
import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.testcontainers.containers.Network;
import org.testcontainers.containers.output.*;
import org.testcontainers.containers.wait.strategy.Wait;
import org.testcontainers.utility.DockerImageName;
import redis.clients.jedis.*;

import java.util.*

import static redis.clients.jedis.Protocol.CLUSTER_HASHSLOTS;

class ScratchJedisClusterWithTestContainers {

  public static void main(String[] args) {
    int i = 0;

    int externalPort = 7379;
    int port = 6379;
    Network network = Network.newNetwork();
    RedisContainer redisContainer = new RedisContainer(DockerImageName.parse("redis:7.0.5"))
        // Use static port binding together with  cluster-announce-port
        //.withCreateContainerCmdModifier(cmd -> cmd.withPortBindings(
        //    new PortBinding(Ports.Binding.bindPort(externalPort), ExposedPort.tcp(port))))
        .withCommand("redis-server --port " + port +
        //    " --cluster-announce-port " + externalPort +
            " --cluster-enabled yes" +
            " --cluster-config-file nodes.conf"+
            " --cluster-node-timeout 5000"+
            " --appendonly yes" +
            " --bind 0.0.0.0" )
        .withNetwork(network)
        .withNetworkMode("bridge")
        .withNetworkAliases("redis-" + i)
        .waitingFor(Wait.forListeningPort());


    // Lambda-based HostAndPortMapper
    HostAndPortMapper nat = hostAndPort -> {
      if (hostAndPort.getPort() == port) {
        return new HostAndPort(redisContainer.getHost(), redisContainer.getMappedPort(port));
      }
      return hostAndPort;
    };

    redisContainer.withLogConsumer((OutputFrame frame) -> System.out.println(frame.getUtf8String()));

    redisContainer.start();

    Set<HostAndPort> hosts = new HashSet<>();
    String redisHost = "127.0.0.1";
    int redisPort = redisContainer.getMappedPort(port+i);
    hosts.add(new HostAndPort(redisHost, redisPort));


    int[] node1Slots = new int[CLUSTER_HASHSLOTS];
    for (int j = 0; j < CLUSTER_HASHSLOTS; j++) {
      node1Slots[j] = j;
    }

    //        Connect to the cluster using Jedis with a password
    DefaultJedisClientConfig.Builder jedisClientConfig = DefaultJedisClientConfig.builder()
    //    .password(redisPassword)
        .hostAndPortMapper(nat)
        .ssl(false)
        .connectionTimeoutMillis(10000)
        .socketTimeoutMillis(4000);

    try(Jedis jedisnode = new Jedis(hosts.iterator().next(),jedisClientConfig.build())){
      jedisnode.clusterAddSlots(node1Slots);
      // await cluster to be ready
      awaitClusterReady(jedisnode, 10, 1000);
      System.out.println(jedisnode.clusterInfo());
      System.out.println(jedisnode.clusterNodes());
    }

    GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
    poolConfig.setMaxTotal(Runtime.getRuntime().availableProcessors());
    poolConfig.setMaxIdle(Runtime.getRuntime().availableProcessors());
    poolConfig.setMinIdle(Runtime.getRuntime().availableProcessors());
    poolConfig.setMaxWaitMillis(2000);



    System.out.println("Hosts " + hosts);
    try (JedisCluster jedis = new JedisCluster(hosts, jedisClientConfig.build(), 3, poolConfig)) {

      Map<String, ConnectionPool> nodes = jedis.getClusterNodes();
      //System.out.println("Connected cluster nodes: " + nodes);
      nodes.forEach((key, value) -> System.out.println(key));
      jedis.set("key", "value"); // This is where the error is seen
      System.out.println("Key set in Redis Cluster: " + jedis.get("key"));
    }
    
  }

  private static void awaitClusterReady(Jedis jedis, int maxAttempts, int delayMillis) {
    for (int attempt = 1; attempt <= maxAttempts; attempt++) {
      String clusterInfo = jedis.clusterInfo();
      if (clusterInfo.contains("cluster_state:ok") && clusterInfo.contains("cluster_slots_assigned:16384")) {
        System.out.println("Cluster is ready!");
        return;
      }
      System.out.println("Attempt " + attempt + ": Cluster not ready. Retrying...");
      try {
        Thread.sleep(delayMillis); // Wait before the next attempt
      } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
        throw new RuntimeException("Sleep interrupted", e);
      }
    }
    throw new RuntimeException("Cluster not ready after " + maxAttempts + " attempts.");
  }
}

Upvotes: 1

Related Questions