CODEWITHSUNDEEP
azure.net-coreazure-cosmosdbpartitioningazure-cosmosdb-sqlapi
Josh L
Josh L

Reputation: 1492

Overcoming CosmosDB 20GB logical partition size

Currently storing telemetry data in CosmosDB from several customers (corporations). Due to certain security compliances, we must at least adhere to "logical" partitioning of customer data. So naturally, customer "corporationId" which is a guid, made the most sense for our partition key. In our application code, any time we perform a query we must always include the partition key, this ensures we don't accidently query data between customers (and improves query performance).

CosmosDB currently has a 20GB partition size limit. For 99% of the corps we work with, they will never come close to this limit, as we are planning on archiving the data after 6 months. One particular corp produces exponentially more data than all the others and we estimated that we will hit this limit after only 60 days.

My Questions

  1. Will cosmosdb simply stop inserting data after the 20gb logical partition is reached or will it separate it into a second partition? I would be okay with a slight performance hit.
  2. Of course we could use /id as a partition key, or something else unique, but this would break our logical partitioning by customer... Has anyone else come up with a clever way of partitioning by customerId and another field combination? Other suggestions

For clarification, this is a .NET Core application and we are using the .NET Core CosmosDB SDK using the SQL api

Upvotes: 2

Views: 1992

Answers (1)

Matias Quaranta
Matias Quaranta

Reputation: 15603

The answer for 1 is that you will start to get failures with StatusCode 403, substatus 1014, as documented here: https://learn.microsoft.com/azure/cosmos-db/sql/troubleshoot-forbidden#partition-key-exceeding-storage

For 2, the comments already mention potential alternatives like the hierarchical partition keys. There is no way at the moment to increase the limit.

Upvotes: 3

Related Questions