CODEWITHSUNDEEP
google-cloud-datastoreapp-engine-flexible
Alex
Alex

Reputation: 1121

Missing entities after insertion in Google Cloud DataStore

After inserting 29447 entities of a single kind in Google Cloud DataStore I wait about 30 seconds and go and check how many entities are there for that particular kind. The surprising thing is that I notice some of them missing (getCurrentKeys returns a bit less than 29447 entities). When I check after a longer period of time (~1 hour), I can then see that all of the entities are there (getCurrentKeys returns the expected 29447 entities).

The code used to read the number of entities is the following:

const runQuery = (query) => {
  return new Promise((resolve, reject) => {
    datastore.runQuery(query)
      .then(results => {
        const entities = results[0];
        resolve(entities);
      })
      .catch(e => reject(e));
  });
};

const getCurrentKeys = () => {
  const query = datastore.createQuery(KIND)
    .select('__key__');
  return runQuery(query);
};

async function main() {
  const currentKeys = await getCurrentKeys();
  console.log(`currentKeys: ${currentKeys.length}`);
}

main();

Any ideas of what could be happening?

Thanks in advance

Upvotes: 0

Views: 205

Answers (2)

Joshua Melcon
Joshua Melcon

Reputation: 249

Non ancestor queries are eventually consistent. It will take a while before all the rows will show up.

This article should explain more:

https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/

Upvotes: 1

Alex
Alex

Reputation: 1121

After a bit more research I think it might be related to the indexes. I believe the indexes aren't getting updated fast enough by the time I run the query. The entities have many properties, so there are many indexes involved.

Upvotes: 0

Related Questions