Configure auto reload template and enable bytecode cache for jinja2 in appengine

Question

How to configure jinja2 in Appengine to:

1. Auto reload when template is updated.
2. Enable bytecode cache, so it can be share among each instances. I prefer jinja2 to produce bytecode when compiling template, and store it to datastore. So next instance will load bytecode instead of repeatedly compile the template.

Answer 1

I have added the bcc like this, using the app engine memcache Client()::

loader = dynloaders.DynLoader()     # init Function loader
bcc = MemcachedBytecodeCache(memcache.Client(), prefix='jinja2/bytecode/', timeout=None)
return Environment(auto_reload=True, cache_size=100, loader=FunctionLoader(loader.load_dyn_all),
                   bytecode_cache=bcc)

My function loader:

def html(self, cid):

    def _html_txt_up_to_date():  # closure to check if template is up to date

        return CMSUpdates.check_no_update(cid, template.modified)

    template = ndb.Key('Templates', cid, parent=self.parent_key).get()
    if not template:
        logging.error('DynLoader (HTML/TXT): %s' % cid)
        return None              # raises TemplateNotFound exception

    return template.content, None, _html_txt_up_to_date

The template model uses template.modified : ndb.DateTimeProperty(auto_now=True)

The closure function:

class CMSUpdates(ndb.Model):                                                                        
    updates = ndb.JsonProperty()

    @classmethod
    def check_no_update(cls, cid, cid_modified):                                                      

        cms_updates = cls.get_or_insert('cms_updates', updates=dict()).updates
        if cid in cms_updates:   # cid modified has dt microseconds
            if cid_modified >= datetime.strptime(cms_updates[cid], '%Y-%m-%d %H:%M:%S'):
                if (datetime.now() - timedelta(days=1)) > cid_modified:
                    del cms_updates[cid]
                    cls(id='cms_updates', updates=cms_updates).put_async()
                return True
            return False         # reload the template
        return True                    

Answer 2

Been few weeks i looking for the solution. And finally i figured it out, i would like to share my code for everyone. There are 4 python source files in my code.

TemplateEngine.py, ContentRenderer.py, TestContent.py & Update_Template.py

File: TemplateEngine.py

Note:

i use now = datetime.utcnow() + timedelta(hours=8) because my timezone is GMT+8

You must use ndb.BlobProperty to store the bytecode, ndb.TextProperty will not work!

from google.appengine.ext import ndb
from datetime import datetime,timedelta

class SiteTemplates(ndb.Model):
  name = ndb.StringProperty(indexed=True, required=True)
  data = ndb.TextProperty()
  uptodate = ndb.BooleanProperty(required=True)

class SiteTemplateBytecodes(ndb.Model):
  key = ndb.StringProperty(indexed=True, required=True)
  data = ndb.BlobProperty(required=True)
  mod_datetime = ndb.DateTimeProperty(required=True)

class LocalCache(jinja2.BytecodeCache):

  def load_bytecode(self, bucket):
    q = SiteTemplateBytecodes.query(SiteTemplateBytecodes.key == bucket.key)
    if q.count() > 0:
      r = q.get()
      bucket.bytecode_from_string(r.data)

  def dump_bytecode(self, bucket):
    now = datetime.utcnow() + timedelta(hours=8)
    q = SiteTemplateBytecodes.query(SiteTemplateBytecodes.key == bucket.key)
    if q.count() > 0:
      r = q.get()
      r.data = bucket.bytecode_to_string()
      r.mod_datetime = now
    else:
      r = SiteTemplateBytecodes(key=bucket.key, data=bucket.bytecode_to_string(), mod_datetime=now)
    r.put()

def Update_Template_Source(tn, source):
  try:
    q = SiteTemplates.query(SiteTemplates.name == tn)
    if q.count() == 0:
      u = mkiniTemplates(name=tn, data=source, uptodate=False)
    else:
      u = q.get()
      u.name=tn
      u.data=source
      u.uptodate=False
    u.put()
    return True
  except Exception,e:
    logging.exception(e)
    return False

def Get_Template_Source(tn):
  uptodate = False

  def Template_Uptodate():
    return uptodate

  try:
    q = SiteTemplates.query(SiteTemplates.name == tn)
    if q.count() > 0:
      r = q.get()
      uptodate = r.uptodate

      if r.uptodate == False:
        r.uptodate=True
        r.put()

      return r.data, tn, Template_Uptodate
    else:
      return None
  except Exception,e:
    logging.exception(e)
    return None

File: ContentRenderer.py

Note: It is very important to set cache_size=0, otherwise bytecode cache function will be disable. I have no idea why.

from TemplateEngine import Get_Template_Source
import jinja2 

def Render(tn,tags):
  global te
  return te.Render(tn, tags)

bcc = LocalCache()
te = jinja2.Environment(loader=jinja2.FunctionLoader(Get_Template_Source), cache_size=0, extensions=['jinja2.ext.autoescape'], bytecode_cache=bcc)

File: Update_Template.py

Note: Use Update_Template_Source() to update template source to datastore.

from TemplateEngine import Update_Template_Source
template_source = '<html><body>hello word to {{title}}!</body></html>'
if Update_Template_Source('my-template.html', template_source):
  print 'template is updated'
else:
  print 'error when updating template source'

File: TestContent.py

Note: Do some test

from ContentRenderer import Render
print Render('my-template.htmnl', {'title':'human'})
'hello world to human!'

You will realize, even you have more than 20 instances in your application, the latency time will not increase, even you update your template. And the template source will update in 5 to 10 seconds.