CODEWITHSUNDEEP
performancegoogle-app-enginescalability
Saad Attieh
Saad Attieh

Reputation: 1486

can you have a large number of sharded counters i.e. large number of entity kinds?

My question is on sharded counters and whether you could have too many. note the below is just a made up example.

Say you want to keep a hit count of different pages on your site. So to prevent datastore contention you decide to shard the hit counters for each page. Now the number of pages grows, hence the number of sharded counters grow.

Assuming you are following the typical sharded examples, each sharded counter has its own kind allowing a query to be built that retrieves all entries blowing to a kind i.e. all entities belonging to that particular sharded counter.

My questions are:

  1. Will a large number of counters (not shards per counter) ) affect performance as there will be so many entity kinds?

  2. Is this the best practise? I mean it looks ugly in the datastore viewer when you have loads of entity kinds as each kind is a sharded counter for a page on your site.

  3. If the above is not good, what would be a better solution?

Upvotes: 1

Views: 131

Answers (1)

MeLight
MeLight

Reputation: 5565

If you followed what you call the "typical shard counter" examples, you can see that there's only one counter type, but you can create different string keys to count different things.

That way you have only one ShardCounter type in your db, but many-many instances with different string keys.

We have a system similar to what you've described. Using only one type of counter we count more than a hundred event types, summing up to around million hits a day. So it's safe to assume that it's pretty scalable ;)

EDIT added counter code examples from Google's documentation:
In the last example you will a counter that has a SHARD_KEY_TEMPLATE variable at the top of the code. This last example allows having different counters with the same shard class.
https://cloud.google.com/appengine/articles/sharding_counters?hl=en

Upvotes: 1

Related Questions