CODEWITHSUNDEEP
pythondjangoapachefastcgi
Li Haoyi
Li Haoyi

Reputation: 21

Exactly how long do Django/Python/FastCGI Processes last?

I have been working on a website in Django, served using FCGI set up using an autoinstaller and a custom templating system.

As i have it set up now, each View is an instance of a class, which is bound to a template file at load time, and not time of execution. That is, the class is bound to the template via a decorator:

@include("page/page.xtag")                    # bind template to view class
class Page (Base):          
    def main(self):                           # main end-point to retrieve web page    
        blah = get_some_stuff()
        return self.template.main(data=blah)  # evaluates template using some data

One thing i have noticed is that since FCGI does not create a new process and reload all the modules/classes every request, changes to the template do not automatically appear on the website until after i force a restart (i.e. by editing/saving a python file).

The web pages also contain lots of data that is stored in .txt files in the filesystem. For example, i will load big snippets of code from separate files rather than leaving them in the template (where they clutter it up) or in the database (where it is inconvenient to edit them). Knowing that the process is persistent, i created an ad-hoc memcache by saving the text i loaded in a static dictionary in one of my classes:

class XLoad:
    rawCache = {} #{name : (time, text)}
    @staticmethod
    def loadRaw(source):
        latestTime = os.stat(source).st_mtime
        if source in XLoad.rawCache.keys() and latestTime < XLoad.rawCache[source][0]:
             # if the cached version of file is up to date, use it
             return XLoad.rawCache[source][1]
        else:
             # otherwise read it from disk, dump it in cache and use that
             text = open(source).read()
             XLoad.rawCache[source] = (latestTime, text)
             return text

Which sped everything up considerably, because the two dozen or so code-snippets which i was loading one-by-one from the filesystem were now being taken directly from the process' memory. Every time i forced a restart, it would be slow for one request while the cache filled up then become blazing fast again.

My question is, what exactly determines how/when the process gets restart, the classes and modules reloaded, and the data i keep in my static dictionary purged? Is it dependent on my installation of Python, or Django, or Apache, or FastCGI? Is it deterministic, based on time, on number of requests, on load, or pseudo-random? And is it safe to do this sort of in-memory caching (which really is very easy and convenient!), or should i look into some proper way of caching these file-reads?

Upvotes: 2

Views: 451

Answers (2)

Sebastian
Sebastian

Reputation: 6057

In addition to the already mentioned reasons, Apache can be configurated to terminate idle fcgi processes after a specified timespan.

Upvotes: 0

agf
agf

Reputation: 176800

It sounds like you already know this.

  1. When you edit a Python file.
  2. When you restart the server.
  3. When there is a nonrecoverable error. Also known as "only when it has to".

Caching like this is fine -- you're doing it whenever you store anything in a variable. Since the information is read only, how could this not be safe? Try not to write changes to a file right after you've restarted the server; but the worst thing that could happen is one page view gets messed up.

There is a simple way to confirm all this -- logging. Have your decorators log when they are called, and log when you have to load a file from disk.

Upvotes: 1

Related Questions