CODEWITHSUNDEEP
azure-cosmosdb
nciao
nciao

Reputation: 601

CosmosDB ChangeFeedProcessor: Can I use 2 separate ChangeFeedProcessor hosts for processing the same unpartitioned feed?

I currently have 2 separate microservices monitoring the same unpartitioned CosmosDB collection (let's call it MasterCollection).

I only ever want one microservice to process MasterCollection's change feed at any given time - the rationale for having 2 separate hosts monitor the same feed basically boils down to redundancy.

This is what I'm doing in code (note that only the first hostName param differs - every other param is identical):

microservice #1:

ChangeFeedEventHost host = new ChangeFeedEventHost("east", monitoredCollInfo, leaseCollInfo, feedOptions, feedHostOptions);

microservice #2:

ChangeFeedEventHost host = new ChangeFeedEventHost("west", monitoredCollInfo, leaseCollInfo, feedOptions, feedHostOptions);

My testing seems to indicate that this works (only one of them processes the changes), but I was wondering if this is a good practice?

Upvotes: 1

Views: 300

Answers (1)

Matias Quaranta
Matias Quaranta

Reputation: 15603

The Change Feed Processor library has a load balancing mechanism that will share the load between them as explained here.

Sharing the load means that they will distribute the Leases among themselves. Each Lease represents a Partition Key Range in the collection.

In your scenario, where you are creating 2 hosts for the same monitored collection using the same lease collection, the effect is that they will share the load and each will hold half of the leases and process changes only for those Partition Key Ranges. So half of the partitions will be processed by the host named westand half by the one named east. If the collection is Single Partition, one of them will process all the changes while the other sits doing nothing.

If what you want is for both to process all the changes independently, you have multiple options:

  1. Use a different lease collection for each Host.
  2. Use the LeasePrefix option in the ChangeFeedHostOptions that can be set on the Host creation. That will let you share the Lease collection for both hosts but they will track independently. Just keep in mind that the RU usage in the Lease collection will raise depending on the amount of activity your main collection has.

Upvotes: 1

Related Questions