CODEWITHSUNDEEP
azureazure-cosmosdb
Mikhail Shilkov
Mikhail Shilkov

Reputation: 35154

Multiple consumers of Cosmos DB Change Feed

I'm using Change Feed Processor library (or Azure Functions Cosmos DB trigger) to subscribe to collection updates. How do I set up multiple independent (non-competing) consumers to the feed of the same collection?

One way is to use multiple lease collections, e.g. leases1, leases2 etc. But that is a bit wasteful.

Is there a way to do that with just one lease collection? (e.g. by specifying a consumer group name somewhere, similar to Event Hubs Processor)

Upvotes: 14

Views: 4532

Answers (2)

Chief Wiggum
Chief Wiggum

Reputation: 2934

You can define a leaseCollectionPrefix for an Azure Function Cosmos DB Trigger. In the Azure portal just click on your function, then on Integrate, then on Advanced editor, which will open your function.json. There you can define the property on your trigger, e.g.

"bindings": [
    {
      "type": "cosmosDBTrigger",
      "name": "documents",
      "direction": "in",
      "leaseCollectionName": "leases",
      "connectionStringSetting": "myDatabase_DOCUMENTDB",
      "databaseName": "myDbName",
      "collectionName": "myCollectionName",
      "createLeaseCollectionIfNotExists": false,
      "leaseCollectionPrefix": "myFunctionSpecificValue"
    }

Additional settings are documented under Documentation:

The following settings customize the internal Change Feed mechanism and Lease collection usage, and can be set in the function.json in the Advanced Editor with the corresponding property names:

  • leaseCollectionPrefix : When set, it adds a prefix to the leases created in the Lease collection for this Function, effectively allowing two separate Azure Functions to share the same Lease collection by using different prefixes.
  • feedPollDelay : When set, it defines, in milliseconds, the delay in between polling a partition for new changes on the feed, after all current changes are drained. Default is 5000 (5 seconds).
  • leaseAcquireInterval : When set, it defines, in milliseconds, the interval to kick off a task to compute if partitions are distributed evenly among known host instances. Default is 13000 (13 seconds).
  • leaseExpirationInterval : When set, it defines, in milliseconds, the interval for which the lease is taken on a lease representing a partition. If the lease is not renewed within this interval, it will cause it to expire and ownership of the partition will move to another instance. Default is 60000 (60 seconds).
  • leaseRenewInterval : When set, it defines, in milliseconds, the renew interval for all leases for partitions currently held by an instance. Default is 17000 (17 seconds).
  • checkpointFrequency : When set, it defines, in milliseconds, the interval between lease checkpoints. Default is always after a successful Function call.
  • maxItemsPerInvocation : When set, it customizes the maximum amount of items received per Function call.

Upvotes: 9

Jesse Carter
Jesse Carter

Reputation: 21207

I've noticed some inconsistencies between consuming the change feed through the Change Feed Processor Library directly vs through the Functions integration.

When using the Change Feed Processor Library, documents like this are generated:

{
    "id": "somegraph.documents.azure.com_obtRAA==_obtRAJvr8AU=..0",
    "_etag": "\"47006e54-0000-0000-0000-59d4fdf20000\"",
    "state": 2,
    "PartitionId": "0",
    "Owner": "CosmosChangeIngestionServiceType",
    "ContinuationToken": "\"143641\"",
    "SequenceNumber": 3322,
    "_rid": "obtRAIhO1RIFAAAAAAAAAA==",
    "_self": "dbs/obtRAA==/colls/obtRAIhO1RI=/docs/obtRAIhO1RIFAAAAAAAAAA==/",
    "_attachments": "attachments/",
    "_ts": 1507130866
}

Ones generated from Functions suspiciously omit the Owner property and set it to null. My understanding was that this Owner field differentiates the change feed consumer and would allow multiple consumers to track progress in the same Lease collection (which would obviously be ideal). So I'm not sure if it's a bug or something I missed when setting up the Function binding but it seems like currently you can only have one Function consumer per lease collection.

UPDATE:

Just had a weekly call with the Cosmos team and asked them this specific question as well as what the status of other lease storage providers such as Table Storage was. They're supposed to be getting back to us by end of day with some clarifications. I will update further when we get back the official information.

Upvotes: 2

Related Questions