CODEWITHSUNDEEP
google-app-enginedatabase-designgoogle-cloud-platformnosqlgoogle-cloud-datastore
N S
N S

Reputation: 2603

Efficiently modelling a Feed schema on Google Cloud Datastore?

I'm using GCP/App Engine to build a Feed that returns posts for a given user in descending order of the post's score (a modified timestamp). Posts that are not 'seen' are returned first, followers by posts where 'seen' = true.

When a user creates a post, a Feed entity is created for each one of their followers (i.e. a fan-out inbox model)

Will my current index model result in an exploding index and/or contention on the 'score' index if many users load their feed simultaneously?

index.yaml
indexes:
- kind: "Feed"
  properties:
  - name: "seen" // Boolean
  - name: "uid" // The user this feed belongs to
  - name: "score" // Int timestamp
    direction: desc

// Other entity fields include: authorUid, postId, postType

A user's feed is fetched by:

SELECT postId FROM Feed WHERE uid = abc123 AND seen = false ORDER BY score DESC

Would I be better off prefixing the 'score' with the user id? Would this improve the performance of the score index? e.g. score="{alphanumeric user id}-{unix timestamp}"

From the docs:

You can improve performance with "sharded queries", that prepend a fixed length string to the expiration timestamp. The index is sorted on the full string, so that entities at the same timestamp will be located throughout the key range of the index. You run multiple queries in parallel to fetch results from each shard.

With just 4 entities I'm seeing 44 indexes which seems excessive.

enter image description here

Upvotes: 0

Views: 219

Answers (1)

Dan Cornilescu
Dan Cornilescu

Reputation: 39824

You do not have an exploding indexes problem, that problem is specific to queries on entities with repeated properties (i.e properties with multiple values) when those properties are used in composite indexes. From Index limits:

The situation becomes worse in the case of entities with multiple properties, each of which can take on multiple values. To accommodate such an entity, the index must include an entry for every possible combination of property values. Custom indexes that refer to multiple properties, each with multiple values, can "explode" combinatorially, requiring large numbers of entries for an entity with only a relatively small number of possible property values. Such exploding indexes can dramatically increase the storage size of an entity in Cloud Datastore, because of the large number of index entries that must be stored. Exploding indexes also can easily cause the entity to exceed the index entry count or size limit.

The 44 built-in indexes are nothing more than the indexes created for the multiple indexed properties of your 4 entities (probably your entity model has about 11 indexed properties). Which is normal. You can reduce the number by scrubbing your model usage and marking as unindexed all properties which you do not plan to use in queries.

You do however have the problem of potentially high number of index updates in a short time - when a user with many followers creates a post with all those indexes falling in a narrow range - hotspots, which the article you referenced applies to. Pre-pending the score with the follower user ID (not the post creator ID, which won't help as the same number of updates on the same index range will happen for one use posting event regardless of sharding being used or not) should help. The impact of followers reading the post (when the score properly is updated) is less impactful since it's less likely for all followers to read the post exactly in the same time.

Unfortunately prepending the follower ID doesn't help with the query you intend to do as the result order will be sorted by follower ID first, not by timestamp.

What I'd do:

  • combine the functionality of the seen and score properties into one: a score value of 0 can be used to indicate that a post was not yet seen, any other value would indicate the timestamp when it was seen. Fewer indexes, fewer index updates, less storage space.
  • I wouldn't bother with sharding in this particular case:
    • reading a post takes a bit of time, one follower reading multiple posts won't typically happen fast enough for the index updates for that particular follower to be a serious problem. In the rare worst case an already read post may appear as unread - IMHO not bad enough for justification
    • delays in updating the indexes for all followers again is IMHO not a big problem - it may just take a bit longer for the post to appear in a follower's feed

Upvotes: 1

Related Questions