CODEWITHSUNDEEP
azure-cosmosdbazure-cosmosdb-changefeed
Dadwals
Dadwals

Reputation: 1321

Facing issue with Azure cosmos change feed processor for Multi region AKS

We have our custom Change Feed processor deployed in single Region in AKS with 5 instances. Things were always running in single region fine. (Please note that each pod instance (feed processor) is assigned a unique [.WithInstanceName(new GUID)].

We recently moving to a multi region setup as following:

Now with the above setup, the result is not very consistent as sometimes after the AKS service deployment our feedprocessor stops recieving the events for some of the collections).

To fix this we need to eventually delete the lease collection and then everything starts working again.

We cannot go live with the above workaround, so need help to resolve the issue.

Here is the code snippet:

Container leaseContainer = cosmosClient.GetContainer(databaseName, leaseContainerName);
            changeFeedProcessor = cosmosClient.GetContainer(databaseName, sourceContainerName)
                .GetChangeFeedProcessorBuilder(processorName: sourceContainerName, async (IReadOnlyCollection<TContainer> changes, CancellationToken cancellationToken) => await onChangesDelegate(changes, cancellationToken))
                    .WithInstanceName($"{Guid.NewGuid()}")
                    .WithLeaseContainer(leaseContainer)
                    .Build();

where leaseContainerName = "container-lease"

Upvotes: 0

Views: 667

Answers (1)

Matias Quaranta
Matias Quaranta

Reputation: 15603

The problem is that you are mixing instances. If you want each region to work independently (the same document change go to both groups of processors independently), set a different processorName.

When you define a cluster of machines with a particular processorName, lease and monitored containers, you define a Deployment Unit. The change feed events are distributed across those machines.

If you deploy 2 clusters but with the same values, then the 10 pods are now the same Deployment Unit, so the changes are spread across the 10 pods, meaning now that a particular change will land in 1 of the instances on one of the region but the other region will not see it.

You could set as processorName the region name for example:

Container leaseContainer = cosmosClient.GetContainer(databaseName, leaseContainerName);
            changeFeedProcessor = cosmosClient.GetContainer(databaseName, sourceContainerName)
                .GetChangeFeedProcessorBuilder(processorName: regionName, async (IReadOnlyCollection<TContainer> changes, CancellationToken cancellationToken) => await onChangesDelegate(changes, cancellationToken))
                    .WithInstanceName($"{Guid.NewGuid()}")
                    .WithLeaseContainer(leaseContainer)
                    .Build();

Upvotes: 1

Related Questions