What's the right way to use excludeFromIndexes in an Embedded Entity in Datastore

Question

Datastore will not allow properties over 1500 bytes in size if they are indexed. So if I have an object

{foo : X, bar : Y}

where Y is longer than 1500 characters I can disable indexing on the individual property and store it by means of an array, thus:

[
 { name : 'foo', value: 'X'}, 
 { name: 'bar', value: 'Y', excludeFromIndexes: true}
]

But this does not work if the property is part of an Embedded Entity (i.e. an entity inside a property of another entity). How do I store something like this?

{ foo : X, bar : { baz : Y } }

This does not work:

[
 { name : 'foo', value: 'X'}, 
 { name: 'bar', value: 
    {
        name: 'baz', 
        value: 'Y', 
        excludeFromIndexes: true
    },
 excludeFromIndexes: true}
]

And this either:

[
 { name : 'foo', value: 'X'}, 
 { name: 'bar', value: {'baz', 'Y' }, excludeFromIndexes: true}
]

---

UPDATE: Here it is an example snippet:

const DataStore = require('@google-cloud/datastore');
const datastore = DataStore({projectId});
const foo1 = { name : { forename: 'Dave', surname : 'Tong' }, colour : 'blue'}

const putAndGet = async data => {
  return new Promise(async (resolve, reject) => {
    try {
      const key = await datastore.save({key: datastore.key([Kind]), data: data});
    } catch (err) {
      reject(err);
    }
    const results = [];
    const query = datastore.createQuery(Kind);
    query.runStream()
      .on('error', (error) => {
        reject(new Error(error));
      })
      .on('data', (entity) => {
        results.push(entity);
      })
      .on('end', () => {
        resolve(results);
      });
  });
}

// This will succeed
putAndGet(foo1).then(ret => {
  for (var i = 0; i < ret.length; i++) {
    console.log(ret[i].name.forename + " likes " + ret[i].colour);
  }

  const str = [];
  for (var i = 0; i < 400; i++) str[i] = 'X';

  const foo2 = {name: {forename: str.join('XXX'), surname: 'Tong'}, colour: 'blue'}
  // This will fail
  return putAndGet(foo2);
}).then(ret => {
  for (var i = 0; i < ret.length; i++) {
    console.log(ret[i].name.forename + " likes " + ret[i].colour);
  }
}).catch(err => {
  console.log(err.message);
});

Answer 1

After some testing, I was able to achieve what you want to do. I have also edited your question, in order to make it clearer for future readers of this post, as you refer to "property part of another property", but that is defined in Datastore as an EmbeddedEntity, so I will stick to that name for a clearer explanation.

Embedded entities can have subproperties which are longer than 1500 Bytes, but you must exclude those subproperties explicitly, or an error will show. To do so, you must declare each EmbeddedEntity as:

{
  "properties": {
    "surname": {
      "stringValue": "A long surname which has more than 1500B",
      "excludeFromIndexes": true
    },
    "forename": {
      "stringValue": "David"
    }
  }
}

But declaring the "excludeFromIndexes": true programatically using NodeJS is not too straight-forward. However finally I managed to solve it. The key is the save() function, where you can declare the properties and subproperties that you want to exclude from indexing, like:

datastore.save({key: entity_key, data: entity_data, excludeFromIndexes: ['prop1', 'prop2.subprop1']});

Here I share a small piece of code (which is an MCV from the piece of code that you shared) that works and creates an entity with some properties containing long suproperties:

const DataStore = require('@google-cloud/datastore');

const projectId = "YOUR_PROJECT_ID";
const datastore = DataStore({projectId});

const data = {name: {forename: 'David', surname: '<YOUR_LONG_STRING>'}, colour: 'purple'}

const Kind = "<YOUR_ENTITY_KIND>";

datastore
  .save({key: datastore.key([Kind]), data: data, excludeFromIndexes: ['name.surname']})
  .then(() => {
    console.log(`Entity saved`);
  })
  .catch(err => {
    console.error('ERROR:', err);
  });

Once you run this piece of code, if you inspect the entity in your Datastore dashboard, you will be able to see that your EmbeddedEntity is defined as I shared at the beginning of my answer.