CODEWITHSUNDEEP
pythongoogle-app-enginegoogle-cloud-datastoreapp-engine-ndb
Brian M. Hunt
Brian M. Hunt

Reputation: 83778

Repeated nested properties of StructuredProperty

While the NDB documentation says:

Although a StructuredProperty can be repeated and a StructuredProperty can contain another StructuredProperty, beware: if one structured property contains another, only one of them can be repeated.

It appeared to me that the clearest interpretation of the documentation may be properly written as "if one structured property contains another structured property".

In which case, I would expect something like this to work:

class KeyList(ndb.Model):
  keys = ndb.KeyProperty(repeated=True)

class Collection(ndb.Model):
  lists = ndb.StructuredProperty(KeyList, repeated=True)

However, this fails with the error:

TypeError: This StructuredProperty cannot use repeated=True because its model class (KeyList) contains repeated properties (directly or indirectly).

It appears that one cannot nest any repeated property inside a repeated StructuredProperty.

From the code, this looks to be the intended behaviour.

In which case the development appserver may differ from the documented behaviour. If the documentation as I've read it is correct, then a fix for the condition in the development server giving rise to the above error might be something like:

if modelclass._has_repeated and isinstance(modelclass, StructuredProperty):
   # ...

Should an AppEngine StructuredProperty be able to contain repeated properties (other than a repeated StructuredProperty)?

Issue report is here: https://github.com/GoogleCloudPlatform/appengine-python-vm-runtime/issues/40

EDIT It's worth noting that the golang docs say this:

Slices of structs are valid, as are structs that contain slices. However, if one struct contains another, then at most one of those can be repeated. This disqualifies recursively defined struct types: any struct T that (directly or indirectly) contains a []T.

The implication I take from this is that repeated non-struct properties on repeated structs are not inherently prohibited at the BigTable level, but rather a side-effect of the Python implementation.

Without further corroboration I would not rely on this implication.

Upvotes: 2

Views: 750

Answers (2)

Patrick Costello
Patrick Costello

Reputation: 3626

This is a bug with our documentation --

With StructuredPropertys, you can only have a single layer of repeated properties.

Some background:

ndb handles StructuredPropertys by exploding properties before writing to Datastore. So for example:

class Inner(ndb.Model):
  a = ndb.StringProperty()

class Outer(ndb.Model):
  inner = ndb.StructuredProperty(Inner, repeated=True)

Then if we write: Outer(inner=[Inner(a="1"), Inner(a="2")]), this actually gets written to the Datastore as:

{
  inner.a = ["1", "2"]
}

However, if instead Inner.a were repeated, we could write something like:

 Outer(inner=[Inner(a=["1", "3"]), Inner(a=["2", "4"])])

This would get written to Datastore like:

{
  inner.a : ["1", "3", "2", "4"]
}

Then when we read this, we have no idea how to parse, as all these are valid:

Outer(inner=[Inner(a=["1", "3"]), Inner(a=["2", "4"])])
Outer(inner=[Inner(a=["1", "3", "2"]), Inner(a=["4"])])
Outer(inner=[Inner(a=["1"]), Inner(a=["3"]), Inner(a=["2"]), Inner(a=["4"])])
Outer(inner=[Inner(a=[]), Inner(a=["1", "3", "2", "4"])])

Note that other libraries (for example Objectify) can avoid this issue by storing both the exploded properties for indexing and an unindexed blob of the entire dataset which can maintain the structure for deserialization.

Upvotes: 4

Klonklion
Klonklion

Reputation: 1

The documentation for structured properties clearly states:

Although a StructuredProperty can be repeated and a StructuredProperty can contain another StructuredProperty, beware: if one structred property contains another, only one of them can be repeated. A work-around is to use LocalStructuredProperty, which does not have this constraint (but does not allow queries on its property values).

Upvotes: -1

Related Questions