CODEWITHSUNDEEP
google-app-enginepaginationgoogle-cloud-datastore
achatain
achatain

Reputation: 56

Google Datastore, "IN" query filter and pagination

We have an application running in Google App Engine and storing data in Google Datastore.

For a given Datastore kind, all our entities have a property type.

We are interested in running a query with an IN query filter to fetch multiple types at once, something like:

type in ['event', 'comment', 'custom']

As there are thousands of entities within this kind, pagination is needed.

The problem we are having is that it is a known limitation of the Datastore that queries with "IN" filters do not support cursor.

Are there sensible ways to get around this limitation?

Using offset would be costy and not performant. Also we can't fetch all entities and filter in the client as we are building an API, hence we don't develop the client ourselves.

Any hint would be really appreciated, thanks!

Upvotes: 2

Views: 2563

Answers (1)

Andrei Volgin
Andrei Volgin

Reputation: 41089

IN filter results in individual EQUAL queries for each item on the list. This is why they do not support cursors - in your case, there will be 3 distinct positions in the index after your run the IN query.

Consider instead adding another property to your entity, which will serve as a flag for this type of API call: its value will be "true" if a type is in ['event', 'comment', 'custom'], or "false" otherwise. Maybe this flag may allow you to make "type" property unindexed - that would be an additional benefit.

With this new indexed property you can use a regular EQUAL filter. It will be faster (1 query instead of 3), and you can use cursors for pagination.

Upvotes: 2

Related Questions