Scaling Cosmos DB

Question

I have a few questions about scaling Cosmos DB, I wasnt able to verify using the offical documentation. A partition key like a userid puts every unique userid into one logical partition and this partition is stored in one physical partition. For a collection with like 2500-5000 RU/s cosmos db provides ten physical partitions, each with a throughput of 250-500 RU/s. Please correct me if i was wrong till now. Another point: Im interested in Cosmos DB with DocumentDB-API.

Now the question: How exactly is the automatically scaling of Cosmos db working? I guess they scale the physical partitions in any way but how exactly is this working. If one physical partition reach his throughput, is there any scaling? Will the logical partitions which are consuming much throughput be moved into another physical partition with free throughput? And what happens if one physical partition reaches its storage limit? Will Cosmos db provide more physical partitions? If yes, the maximum throuhput would be splitted into the number of the current physical partitions, is this correct as far?

Answer 1

I'd love someone from the Cosmos DB team to provide an authoritative answer. And although Microsoft points to Stackoverflow for questions, I have not seen official answers. That being said, this is my attempt to answer.

For a collection with like 2500-5000 RU/s cosmos db provides ten physical partitions, each with a throughput of 250-500 RU/s.

Initially there are 10 physical partitions. But this isn't documented anywhere.

If one physical partition reach his throughput, is there any scaling?

No, a partition is split only when the storage limit for a partition is reached (10GB it seems).

Will the logical partitions which are consuming much throughput be moved into another physical partition with free throughput?

No.

And what happens if one physical partition reaches its storage limit? Will Cosmos db provide more physical partitions? If yes, the maximum throuhput would be splitted into the number of the current physical partitions, is this correct as far?

From the documentation: When a physical partition p reaches its storage limit, Cosmos DB seamlessly splits p into two new partitions p1 and p2 and distributes values corresponding to roughly half the keys to each of the partitions. This split operation is invisible to your application.

You didn't mention RU per minute and this is a very important aspect of the system. It seems that RU/m are used as needed in any partition (as opposed to RU/s that are tied to a physical partition). This is specifically mentioned in a Microsoft blog as a way to deal with hot partition keys. Update: The blog post is no longer available; perhaps because the information was not accurate. This was the original blog post:

https://web.archive.org/web/20170609030349/https://azure.microsoft.com/en-us/blog/azure-cosmosdb-case-study-around-ru-min-with-the-universal-store-team/

Answer 2

No, a partition is split only when the storage limit for a partition is reached (10GB it seems).

Can you verifiy the spliting of a full partition into two new partitions? For testcases i spread multiple partitionkey-values across all partitions and filled one partition with a unique partitionkey-value. Related to the cosmos db metrics, this value consumes storage about 5,1 GB and the physical partition has reached its 10 GB storage limit. If i try to add more documents with the same partitionkey-value, my c# console application in VS throws an ForbiddenException which says "Storage quota for 'Document' exceeded"." The metrics view also shows a full partition but there aint any automatic splitting of this full partition. Can anyone help me there whats wrong?

Answer 3

Partition management is fully managed by Azure Cosmos DB, and you do not have to write complex code or manage your partitions. Cosmos DB containers are unlimited in terms of storage and throughput. I would recommend you read this for more information.