CODEWITHSUNDEEP
google-app-enginegoogle-cloud-datastore
Attila O.
Attila O.

Reputation: 16625

App Engine multiple namespaces

Recently there's been some data structure changes in our app, and we decided to use namespaces to separate different versions of of the data, and a mapreduce task that converts old entities to the new format.

Now that's all fine, but we don't want to always isolate the entire data set we have. The biggest part of our data is stored in a kind that's pretty simple and doesn't need to change often. So we decided to use per-kind namespaces.

Something like:

class Author(ndb.model.Model):
    ns = '2'

class Book(ndb.model.Model):
    ns = '1'

So, when migrating to version 2, we don't need to convert all our data (and copy all 'Book' kinds to the other namespace), only entities of the 'Author' kind. Then, instead of defining the appengine_config.namespace_manager_default_namespace_for_request, we just the 'namespace' keyword arguments to our queries:

Author.query(namespace=Author.ns).get()

Question: how to store (i.e. put()) the different kinds using these different namespaces? Something like:

# Not an API
Author().put(namespace=Author.ns)

Of course, the above doesn't work. (Yes, I could ask the datastore for an avaliable key in that namespace, and then use that key to store the instance with, but it's an extra API call that I'd like to avoid.)

Upvotes: 3

Views: 1620

Answers (3)

Brian M. Hunt
Brian M. Hunt

Reputation: 83858

To solve a problem like this I wrote a decorator as follows:

MY_NS = 'abc'

def in_my_namespace(fn):
    """Decorator: Run the given function in the MY_NS namespace"""
    from google.appengine.api import namespace_manager

    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        orig_ns = namespace_manager.get_namespace()
        namespace_manager.set_namespace(MY_NS)

        try:
            res = fn(*args, **kwargs)
        finally: # always drop out of the NS on the way up.
            namespace_manager.set_namespace(orig_ns)

        return res

    return wrapper

So I can simply write, for functions that ought to occur in a separate namespace:

@in_my_namespace
def foo():
   Author().put() # put into `my` namespace

Of course, applying this to a system to get the results you desire is a bit beyond the scope of this, but I thought it might be helpful.

EDIT: Using a with context

Here's how to accomplish the above using a with context:

class namespace_of(object):
    def __init__(self, namespace):
        self.ns = namespace

    def __enter__(self):
        self.orig_ns = namespace_manager.get_namespace()
        namespace_manager.set_namespace(self.ns)

    def __exit__(self, type, value, traceback):
        namespace_manager.set_namespace(self.orig_ns)

Then elsewhere:

with namespace_of("Hello World"):
    Author().put() # put into the `Hello World` namespace

Upvotes: 5

schuppe
schuppe

Reputation: 2033

A Model instance will use the namespace you set with the namespace_manager[1] as you can see here: python/google/appengine/ext/db/init.py

What you could do is create a child class of Model which expects a class-level 'ns' attribute to be defined. This sub class then overrides put() and sets the namespace before calling original put and resets the namespace afterwards. Something like this:

'''
class MyModel(db.Model):

    ns = None

    def put(*args, **kwargs):
        if self.ns == None:
            raise ValueError('"ns" is not defined for this class.')
        original_namespace = namespace_manager.get_namespace()
        try:
            super(MyModelClass, self).put(*args, **kwargs)
        finally:
            namespace_manager.set_namespace(original_namespace)
'''

[1] http://code.google.com/appengine/docs/python/multitenancy/multitenancy.html

Upvotes: 3

Adam Crossland
Adam Crossland

Reputation: 14213

I don't think that it is possible to avoid the extra API call. Namespaces are encoded into the entity's Key, so in order to change the namespace within which a entity is stored, you need to create a new entity (that has a Key with the new namespace) and copy the old entity's data into it.

Upvotes: 0

Related Questions