CODEWITHSUNDEEP
azureazure-functionsazure-cosmosdbazure-cosmosdb-changefeed
CognitiveComplexity
CognitiveComplexity

Reputation: 164

One Azure Function that can listen to Cosmos DB Change Feed for all containers

Right now I have one Cosmos DB that have three different containers, therefore I use three different functions that are listening for Change Feed events from this Cosmos DB.

In the future amount of my containers will be grown from 3 to 100.

So, is it possible to have one function that will be listening for all changes in all containers and that can detect from what container changes have come?

Upvotes: 2

Views: 1607

Answers (2)

Noah Stahl
Noah Stahl

Reputation: 7563

The recommended pattern with Cosmos DB is to have a single or few data containers and partition data logically via property values, rather than segmenting into many containers. If at all possible, for change feed and other reasons, it would be worthwhile to review the proposed design to see if there is a way to consolidate containers and avoid this pain.

That said, if an unknown and growing number of containers must be supported, one way that might be achieved dynamically is with the Change Feed Processor via the SDK. When instantiating a processor instance using GetChangeFeedProcessorBuilder, you can provide the container name as a parameter. Given a configured or discovered list of all target containers, multiple change feed processor instances could be created and run in parallel.

This could be hosted in multiple ways. Consider using an ASP.NET Core app with an IHostedService, and avoiding Azure Functions in this case.

Upvotes: 1

rickvdbosch
rickvdbosch

Reputation: 15571

In short: no.

Change feed in Azure Cosmos DB is a persistent record of changes to a container in the order they occur. Change feed support in Azure Cosmos DB works by listening to an Azure Cosmos container for any changes.

and

Change feed is available for each logical partition key within the container, and it can be distributed across one or more consumers for parallel processing.

The documentation on Change feed in Azure Cosmos DB clearly states a Change Feed is for one specific container.

There probably are, however, different approaches you could take to solve your problem. Most important question is: what actually is the problem you are trying to solve?

If you need to process the changes in Cosmos DB using a Function, I can imaging the logic for processing the changes can (will) be different for each type of data. So for each container. If this is not the case the data doesn't have to be in different containers?

One option could be to create a timer triggered Function that will be Reading change feed with a pull model. This enables you to loop the containers in that Function and prepare processing the changes per container (for instance by putting the information in a queue or using Durable Functions with the Fan-Out/Fan-In pattern).

Upvotes: 1

Related Questions