CODEWITHSUNDEEP
google-app-enginegoogle-cloud-datastore
Eric Wei
Eric Wei

Reputation: 151

How to implement "Sharding Counters" to create or update single entity more than 5 writes per second in Cloud Datastore?

I need to improve the server's performance by increasing the writing throughput in Google Cloud Datastore.

Requirement:

When the server gets more than 5 requests to create the user data at the same time, the server needs to create or update those entities.

However, I encountered a writing contention problem.

I know a possible solution is to use a write-behind cache mechanism moving the writes operation that can lead to contention to Memcache and a Taskqueue slowing down the Cloud Datastore hit rate.

But I want to do it in parallel without any delay time.

1.Is it possible to apply "Sharding Counters" to create or update ndb's user model?

2.Could you provide any sample codes for this?

Upvotes: 3

Views: 882

Answers (1)

Eric Wei
Eric Wei

Reputation: 151

I did some research and want to share as below https://weishihhsun.blogspot.com/search/label/Google%20App%20Engine

The key point is we can do sharding on our entity group with a random unique id as shown below. 

NUM_SHARDS = 1000

shard_string_index = str(random.randint(0, NUM_SHARDS - 1))

FriendShip(id=shard_string_index,
           user_key='user Id', 
           friend_key='frind Id')

To simultaneously write a thousand of Friendship entities in parallel, we just need to set the number of NUM_SHARDS as 1000.

Upvotes: 0

Related Questions