CODEWITHSUNDEEP
google-cloud-platformcloudgoogle-cloud-datastoredatastore
Hanming Zeng
Hanming Zeng

Reputation: 367

Save Large json to google cloud datastore

I want to save some large json to datastore, where len(json)>=80000000 (80MB) but I am getting: ServiceUnavailable: 503 413:Request Entity Too Large

I could potentially save it in cloud storage instead, but I guess I will lose indexing and faster querying ability from datastore. What's the best solution here?

def save_serialized_data_to_db(json, name):
  datastore_client = datastore.Client()

  kind = 'SerializedData'
  serialized_data_key = datastore_client.key(kind, name)

  serialized_data = datastore.Entity(key=serialized_data_key)
  serialized_data['json'] = json

  datastore_client.put(serialized_data) // getting: ServiceUnavailable: 503 413:Request Entity Too Large

  return serialized_data

Upvotes: 1

Views: 935

Answers (2)

guillaume blaquiere
guillaume blaquiere

Reputation: 75715

In my company with perform a special process

  • Extract from the full JSON, the indexable and searchable values.
  • Create a new JSON with those values.
  • Store the full JSON into Cloud Storage
  • Store the new JSON + the Cloud Storage file path into Datastore.

We are also looking at using MongoDB. I haven't feedback for you, yet.

Upvotes: 1

Gabe Weiss
Gabe Weiss

Reputation: 3332

Datastore has a max transaction size of 10MB.

The limits can be found here: https://cloud.google.com/datastore/docs/concepts/limits

Relevant cut/paste:

Maximum API request size. This limit applies when Datastore mode is used outside of Google App Engine. If Datastore mode is used from App Engine, the limit depends on the client library that is used. 10 MiB

Maximum size for a transaction 10 MiB

Maximum size for an entity 1,048,572 bytes (1 MiB - 4 bytes)

Upvotes: 0

Related Questions