CODEWITHSUNDEEP
google-cloud-datastore
zono
zono

Reputation: 8584

Google Cloud datastore pagination

We store data in Google Cloud Datastore. We want to provide APIs to our users. Our APIs's pagination specification is header based same as github API. We want users use page parameter.

pagination specification

e.g.

Link: <https://api.github.com/search/code?q=addClass+user%3Amozilla&page=15>; rel="next",
  <https://api.github.com/search/code?q=addClass+user%3Amozilla&page=34>; rel="last",
  <https://api.github.com/search/code?q=addClass+user%3Amozilla&page=1>; rel="first",
  <https://api.github.com/search/code?q=addClass+user%3Amozilla&page=13>; rel="prev"

https://developer.github.com/guides/traversing-with-pagination/

API

End-Users <--- Backend Code (PHP) <--- Google Cloud Datastore

As you know Google Cloud Datastore recommends using cursor for improving performance and cost. But we don't want end-users use cursor. Is it possible that end-users use integer page number instead of cursor and cursor is used in backend?

We use Google's PHP client.

https://github.com/google/google-api-php-client-services

Upvotes: 4

Views: 6757

Answers (3)

kinhong
kinhong

Reputation: 36

I am unsure if GCloud datastore's cursor can be used in pagination implementations where a user needs to "jump" to different page numbers. Here is the way I implement datastore pagination in my project:

Given an existing query (sorted, filtered, etc.), an "offset" (a number) and a "limit" (a number)

  1. Run a key-only query with limit (offset+limit)
  2. Slice the results in step 1. from the offset (until the end) and map them to Datastore.KEY
  3. Create and run a datastore.get() query with the set of keys obtained in step 2.

Code in NodeJS:

async function paginate(query, offset, limit) {
  let results;
  if (offset > 0) {
    // Get the keys (low cost)
    query.select('__key__');
    query.limit(offset + limit);
    const [keys] = await datastore.runQuery(query);

    const [entities] = await datastore.get(keys.slice(offset).map(k => k[Datastore.KEY]));
    return entities;
  }
  query.limit(limit);
  const [entities] = await datastore.runQuery(query);
  return entities;
}

Upvotes: 1

deepakssn
deepakssn

Reputation: 5363

Google recommends usage of cursor and has an example on their documentation. https://cloud.google.com/datastore/docs/concepts/queries#cursors_limits_and_offsets

const pageSize = 5
query := datastore.NewQuery("Tasks").Limit(pageSize)
if cursorStr != "" {
        cursor, err := datastore.DecodeCursor(cursorStr)
        if err != nil {
                log.Fatalf("Bad cursor %q: %v", cursorStr, err)
        }
        query = query.Start(cursor)
}

// Read the tasks.
var tasks []Task
var task Task
it := client.Run(ctx, query)
_, err := it.Next(&task)
for err == nil {
        tasks = append(tasks, task)
        _, err = it.Next(&task)
}
if err != iterator.Done {
        log.Fatalf("Failed fetching results: %v", err)
}

// Get the cursor for the next page of results.
nextCursor, err := it.Cursor()

Upvotes: 4

Renato Byrro
Renato Byrro

Reputation: 3784

I believe you can have something like an OFFSET by using GQL, but such operation will cost you a lot of money (doing the equivalent of LIMIT 1000, 10 will count as 1,010 reads - not just the 10 you actually get back).

Cut down costs of OFFSET for pagination

Let's say your page size is 10 items and a user asks to jump to page 5. You'll need to query for the first 40 entities, get the cursor and run the query again, now providing the cursor and limiting to 10.

It's recommended that, in the first query, you fetch with keys_only=True. This way you can:

Upvotes: 8

Related Questions