CODEWITHSUNDEEP
azureazure-cosmosdb
Trilok Suresh
Trilok Suresh

Reputation: 11

Change Feed Processor - Multiple consumers acquiring leases on all the partitonIds

I have 2 instances of a worker role listening to the same collection in CosmosDb using the Change Feed Processor. I am receiving duplicate data because both the consumers have acquired leases on all the partitions and the same data is being streamed twice.

I have used the LeasePrefix property of the ChangeFeedHostOptions and set it to the instance name (RoleEnvironment.CurrentRoleInstance.Id)

Here is an example from lease collection:

Consumer1
{
    "id": "WorkerRole1_IN_1xyz.documents.azure.com_v0QfAA==_v0QfAKJwMgA=..0",
    "_etag": "\"0000c601-0000-0000-0000-5acf0fca0000\"",
    "state": 2,
    "PartitionId": "0",
    "Owner": "WorkerRole1_IN_1",
    "ContinuationToken": "\"115725\"",
    "SequenceNumber": 1,
    "_rid": "DiADAKjOkAMDAAAAAAAAAA==",
    "_self": "dbs/DiADAA==/colls/DiADAKjOkAM=/docs/DiADAKjOkAMDAAAAAAAAAA==/",
    "_attachments": "attachments/",
    "_ts": 1523519434
}

Consumer2:
{
    "id": "WorkerRole1_IN_0xyz.documents.azure.com_v0QfAA==_v0QfAKJwMgA=..4",
    "_etag": "\"00007e01-0000-0000-0000-5acf0fcc0000\"",
    "state": 2,
    "PartitionId": "0",
    "Owner": "WorkerRole1_IN_0",
    "ContinuationToken": "\"115725\"",
    "SequenceNumber": 1,
    "_rid": "DiADAPNPPAMDAAAAAAAAAA==",
    "_self": "dbs/DiADAA==/colls/DiADAPNPPAM=/docs/DiADAPNPPAMDAAAAAAAAAA==/",
    "_attachments": "attachments/",
    "_ts": 1523519436
}

Here are the feed options I have used:

ChangeFeedOptions feedOptions = new ChangeFeedOptions
{
    StartFromBeginning = true,
    MaxItemCount = 10000,
    StartTime = DateTime.Today.AddDays(-7)
};

ChangeFeedHostOptions feedHostOptions = new ChangeFeedHostOptions
{
    LeasePrefix = Configs.hostName,
    FeedPollDelay = TimeSpan.FromSeconds(30)
};

Can someone tell me if I am missing something?

Upvotes: 1

Views: 817

Answers (1)

Matias Quaranta
Matias Quaranta

Reputation: 15603

The reason you are receiving duplicates is because you are using the LeasePrefix. The LeasePrefix states:

This can be used to support multiple ChangeFeedEventHost instances pointing at the same feed while using the same auxiliary collection.

The goal of that property is to actually do that, share the same lease collection with multiple independent Hosts.

To solve your issue, just remove the LeasePrefix assignment or use the same Prefix for both.

Upvotes: 1

Related Questions