Apache Ignite CacheRebalanceMode Is Not Respected By Nodes

Question

The setup: Hello folks I have a simple Apache Ignite setup with two Ignite instances configured as server nodes over C# and one Ignite instance as a client node over java.

What is the goal: Populate data on instance 1 and instance 2 but avoid movement of data between them. In other words data receiced on each node must stay in node. Then using the java client to run queries against the two nodes either combined (distributed join) or per node (using affinity).

The issue: With one server node everything works as expected, however on more than one server nodes, data of the cluster is balancing between the x member nodes even if I have expliccitly set the CacheRebalanceMode to None which should disable the rebalance between then nodes. The insert time is increase by 4x-10x times, function to each node's populated data.

P.S. I have tried change the cache mode from Partitioned to Local where each node will have isolated the data in it's internal H2 db however in that case the Java client is unable to detect the nodes or read any data from the cache of each node.

Java Client Node

        IgniteConfiguration cfg = new IgniteConfiguration();
        // Enable client mode.
        cfg.setClientMode(true);

        // Setting up an IP Finder to ensure the client can locate the servers.
        TcpDiscoveryMulticastIpFinder ipFinder = new TcpDiscoveryMulticastIpFinder();
        ipFinder.setAddresses(Collections.singletonList("127.0.0.1:47500..47509"));
        cfg.setDiscoverySpi(new TcpDiscoverySpi().setIpFinder(ipFinder));

        // Configure Ignite to connect with .NET nodes
        cfg.setBinaryConfiguration(new BinaryConfiguration()
                        .setNameMapper(new BinaryBasicNameMapper(true))
                        .setCompactFooter(true)

        // Start Ignite in client mode.
        Ignite ignite = Ignition.start(cfg);

        
        IgniteCache<AffinityKey, Trade> cache0 = ignite.cache(CACHE_NAME);
        IgniteCache<AffinityKey, BinaryObject> cache = cache0.withKeepBinary();
    
        // execute some queries to nodes

C# Server Node

               IIgnite _ignite = Ignition.Start(IgniteUtils.DefaultIgniteConfig()));

                // Create new cache and configure queries for Trade binary types.
                // Note that there are no such classes defined.
                var cache0 = _ignite.GetOrCreateCache<AffinityKey, Trade>("DEALIO");

                // Switch to binary mode to work with data in serialized form.
                _cache = cache0.WithKeepBinary<AffinityKey, IBinaryObject>();

               //populate some data ...

        public static IgniteConfiguration DefaultIgniteConfig()
        {
            return new IgniteConfiguration
            {

               
                PeerAssemblyLoadingMode = PeerAssemblyLoadingMode.CurrentAppDomain,
                BinaryConfiguration = new BinaryConfiguration
                {
                    NameMapper = new BinaryBasicNameMapper { IsSimpleName = true },
                    CompactFooter = true,
                    TypeConfigurations = new[] {
                        new BinaryTypeConfiguration(typeof(Trade)) {
                            Serializer = new IgniteTradeSerializer()
                        }
                    }
                },
                DiscoverySpi = new TcpDiscoverySpi
                {
                    IpFinder = new TcpDiscoveryMulticastIpFinder
                    {
                        Endpoints = new[] { "127.0.0.1:47500..47509" }
                    },
                    SocketTimeout = TimeSpan.FromSeconds(0.10)
                },
                Logger = new IgniteNLogLogger(),
                CacheConfiguration = new[]{
                    new CacheConfiguration{
                            PartitionLossPolicy=PartitionLossPolicy.Ignore,
                            RebalanceMode=CacheRebalanceMode.None,
                            Name = CACHE_NAME,
                            CacheMode = CacheMode.Partitioned,
                            Backups = 0,
                            QueryEntities = new[] { 
                                new QueryEntity(typeof(AffinityKey), typeof(Trade))
                            }
                }
            }
            };
        }

Answer 1

Populate data on instance 1 and instance 2 but avoid movement of data between them. In other words data receiced on each node must stay in node.

The problem is, that's not how Ignite works. A cache is really a cluster-wide concept; that is, you put data into the cache, not into a specific node. If you want to keep data on a specific node, you can use the LOCAL cache but you've already seen the downside to that.

Rather than force data onto a specific machine, Ignite uses the concept of "affinity." You tell it how data is related and it will automatically keep those data together.

Putting data into a specific machine works fine until you want to change the number of nodes (or have one fail). By defining affinity, Ignite can scale up and down automatically.

Answer 2

CacheRebalanceMode.None basically means "do not move data to a newly added node", not "keep all entries locally". cache.Put will still send the data to a primary node for a given key.

The following code demonstrates how None mode works:

var cfg = new IgniteConfiguration
{
    AutoGenerateIgniteInstanceName = true,
    CacheConfiguration = new[]
    {
        new CacheConfiguration
        {
            Name = "c",
            RebalanceMode = CacheRebalanceMode.None
        }
    }
};

var ignite1 = Ignition.Start(cfg);
var cache1 = ignite1.GetCache<int, int>("c");

// Put before starting the second node.
cache1.PutAll(Enumerable.Range(1, 10000).Select(x => new KeyValuePair<int, int>(x, x)));

var ignite2 = Ignition.Start(cfg);
var cache2 = ignite2.GetCache<int, int>("c");

Thread.Sleep(5000); // Wait for the rebalance.
Console.WriteLine(cache1.GetLocalSize());
Console.WriteLine(cache2.GetLocalSize()); // 0 with None

With None mode you'll see zero size of the second cache, and with any other mode it won't be zero.