CODEWITHSUNDEEP
google-cloud-datastore
gopi.mails
gopi.mails

Reputation: 21

What can be a datastore entity type to store and query a long value with index

Want to know what can be a best type to store and query a long value in Google Cloud Datastore while creating an entity.

If I try to store a long value as integer, it gets stored, but fails to return when you do a query. It makes sense why query fail. Store and query works good if I store the long int value as string or as floating point number.

Want to understand what can be the best practice for storing such a value as I want to "Index the property".

Upvotes: 0

Views: 587

Answers (2)

Ryan Loggerythm
Ryan Loggerythm

Reputation: 3324

I assume this is your exact problem:

google.api_core.exceptions.InvalidArgument: 400 The value of property "{myVar}" is longer than 1500 bytes.

Even if you've explicitly set the field to not be indexed, or if you've chosen type "text" which has no indexing option, your inserts/upserts will fail for objects over 1500 bytes.

Here's how you can get past that issue:

client = datastore.Client()
kind = 'myKind'
uniqueKey = f'{howeverYouGenerateYourKeys}'
completeKey = client.key(kind, uniqueKey)
# incompleteKey = client.key(kind)  # alternately, you could let Google generate the key id for you!
entity = datastore.Entity(key=completeKey, exclude_from_indexes=['reallyLongField'])
entity.update({
    'firstName': x.firstName,
    'lastName': x.lastName,
    'favoriteDinosaur': x.rawr,      
    'reallyLongField': x.deeeetails
})
client.put(entity)

The important part is adding exclude_from_indexes to datastore.Entity(...).

Ref: https://cloud.google.com/datastore/docs/concepts/indexes

Upvotes: 1

Artsiom Miksiuk
Artsiom Miksiuk

Reputation: 4323

You can find Datastore limits by the following link. https://cloud.google.com/datastore/docs/concepts/limits

Also it will be useful for you if you'll get familiar with this concepts of Datastore indexing. https://cloud.google.com/appengine/docs/standard/python/datastore/indexes

And also link to best practices. https://cloud.google.com/datastore/docs/best-practices

Regarding your approach, to store long integers as a string and still be able to index such property. There is limitation on indexed strings in UTF-8 size in 1500 bytes. In general, this is the only way you can store it like that, but I can propose additional ways to increase this limit.

  1. Split string on separate parts and store as array of strings.
  2. Compress resulting long string.
  3. Make reduced accuracy conversion and store power of number (10^100).

But, any way currently this is the only way I can see it can be done, as you described, in string format.

Upvotes: 2

Related Questions