GAE: how to add new data in a memcache var

Question

I'm blocked right now.

The problem is this: I have my "view" of a list of books stored in memcache. If I add a new book, I want to add the new book to the memcache variable that stores the list of books. Same for update and delete.

You can say: "you can flush the key and gather all data again". But due to eventual consistency, when I add and immediately query the new data, the new data isn't still there.

You can say: "use ancestors to avoid eventual consistency". The books are the root entities, and for performance, it's not a good idea to use ancestors at this level.

So, the idea is to read from datastore the less as possible, and have memcache syncronized.

Now my code, that doesn't work:

class Book(ndb.Model):
    """ A Book """
    title = ndb.StringProperty(required=True)
    author = ndb.StringProperty(required=True)

    @classmethod
    def getAll(cls):
        key = 'books'
        books = memcache.get(key)
        if books is None:
            books = list(Book.query().order(Book.title).fetch(100))
            if not memcache.set(key, books):
                logging.error('Memcache set failed for key %s.', key)
        else:
            logging.info('Memcache hit for key %s.', key)
        return books

    @classmethod
    def addMemcache(cls, newdata):
        mylist = memcache.get('books')
        if mylist:
            mylist.insert(0, newdata)
            if not memcache.set('books', mylist):
                logging.error('Memcache set failed for key books.')

    # This saves the data comming from the form
    @classmethod
    def save(cls, **kwargs):
        book = Book(title=kwargs['title'],
                    author=kwargs['author']
               )
        # Save
        book.put()
        # Add to memcache for this key
        logging.info('Adding to Memcache for key books.')
        cls.addMemcache([book.title, book.author])
        return book

For now I'm simply inserting at start of the list. The problem with my code is that when I ADD to memcache, I'm missing some kind of property because the jinja template says "UndefinedError: 'list object' has no attribute 'title'", when is trying to represent this line:

<td>{{ line.title[:40]|escape }}</td>

Which is obvious, because is a list, not an object with properties. But why it works when I convert the object in a list in the function getAll(), making books = list(the_query)?

My other problems will be how to modify an especific book (in this case I can flush de memcache and read again, because there is not eventual consistency problem I think) and how to delete (how to identify a unique element in the list if 2 books have the same name).

Any suggestion? Or must I change my solution to the problem of have memcache syncronized?

Answer 1

With the help of @DanielRoseman I got the final solution for the problem.

I just want to leave here the complete solution for other interested "stackers". It includes the add, edit and delete elements to memcache, that are working now.

# These classes define the data objects to store in AppEngine's data store.
class Book(ndb.Model):
    """ A Book """
    title = ndb.StringProperty(required=True)
    author = ndb.StringProperty(required=True)
    deleted = ndb.BooleanProperty(default=False)

    MEMCACHE_TIMEOUT = 0

    # Key to use in memcache for the list of all books
    @staticmethod
    def book_memkey(key='book_list'):
        return str(key)

    # Search all
    @classmethod
    def get_all(cls):
        key = cls.book_memkey()
        books = memcache.get(key)
        if books is None:
            books = list(Book.query().order(Book.title).fetch(100))
            if not memcache.set(key, books, cls.MEMCACHE_TIMEOUT):
                logging.error('Memcache set failed for key %s.', key)
        else:
            logging.info('Memcache hit for key %s.', key)
        return books

    # Save a Book and return it
    @classmethod
    def save(cls, **kwargs):
        book = cls(title=kwargs['title'],
                   author=kwargs['author']
                  )
        book.put()
        # Modify memcache for this key
        book.add_to_memcache()
        return book

    # ------------------------
    # Methods for the instance
    # ------------------------

    # Add a new element to memcache
    def add_to_memcache(self):
        data = memcache.get(self.book_memkey())
        if data:
            logging.info('Adding to Memcache for key %s.', self.book_memkey())
            data.insert(0, self)
            if not memcache.set(self.book_memkey(), data, self.MEMCACHE_TIMEOUT):
                logging.error('Memcache set failed for key %s.', self.book_memkey())

    # Remove an element from memcache
    def del_from_memcache(self):
        data = memcache.get(self.book_memkey())
        if data:
            logging.info('Removing from Memcache for key %s.', self.book_memkey())
            try:
                # Search the object in the list
                element = filter(lambda idx: idx.key == self.key, data)[0]
            except IndexError:
                pass
            else:
                logging.info('Removing element %s.', element)
                data.remove(element)
                if not memcache.set(self.book_memkey(), data, self.MEMCACHE_TIMEOUT):
                    logging.error('Memcache set failed for key %s.', self.book_memkey())

    # Update an element on memcache
    def update_memcache(self):
        data = memcache.get(self.book_memkey())
        if data:
            logging.info('Updating Memcache for key %s.', self.book_memkey())
            try:
                # Search the object in the list
                element = filter(lambda idx: idx.key == self.key, data)[0]
            except IndexError:
                pass
            else:
                logging.info('Updating element %s.', element)
                data[data.index(element)] = self
                if not memcache.set(self.book_memkey(), data, self.MEMCACHE_TIMEOUT):
                    logging.error('Memcache set failed for key %s.', self.book_memkey())

    # Update a chapter
    def update(self, **kwargs):
        if 'title' in kwargs:
            self.title = kwargs['title']
        if 'author' in kwargs:
            self.author = kwargs['author']
        # Save
        self.put()
        self.update_memcache()

    # Delete de book (mark as deleted). Optionally you can assign Value=False to undelete
    def virtual_delete(self, value=True):
        self.deleted = value
        if value:
            self.del_from_memcache()
        else:
            self.add_to_memcache()
        self.put()

Answer 2

You're doing two different things when setting your memcache values. In getAll, on a cache miss you do memcache.set(key, books) where books is a list of Book instances. But in addMemcache (called by save) you insert a list of lists, where the inner list is the title and author of the book. So as you've noted, when you get the values from the cache, they are a mixture of instances and lists.

Seems like the line in save should just be:

cls.addMemcache(book)

so that you consistently set Book instances to the cache.

(Also note that I'd probably make addMemcache a normal instance method rather than a classmethod, that adds self to the memcache list. And in save it would be better to instantiate cls rather than explicitly invoking Book, in case you ever subclass.)