How do I go about setting up a TDD development process with Google App Engine?

Question

I'm primarily a Ruby guy, but lately I've been working on a lot of Python stuff, in particular, App Engine code. In Ruby, I'd use automated continuous integration (autotest), code coverage tools (rcov), static analysis (reek), and mutation testing (heckle) in my development process, but I'm not sure how best to set up a similar development process for an App Engine environment. I'd also be interested in analogs to RSpec and Cucumber for Python that could work in App Engine.

Answer 1

For mutation testing try mutmut, it's available at https://github.com/boxed/mutmut/

It's as simple as "pip install mutmut" then "mutmut run" if you use pytest.

Answer 2

It looks like another new option is Agar, which is based on google.appengine.ext.testbed, part of the App Engine SDK. See this blog post for details.

Answer 3

On my GAE project, I use:

• NoseGAE—This is the critical piece that ties all the rest together
• Mock, as in John's excellent answer. I use this largely for AWS and other web services
• Fixtures (the package, not the idea)

I also prefer many of Rails's idioms. I broke my tests into unit, and functional using Python packages. You can run a subset of tests using --tests=unit or --tests=functional. It is all a bit more manual than Rails but at least I can unit test the hard stuff and make sure I never have regressions.

I also made a simple FunctionalTest class to do much of the very common actions in Rails, such as assert_response and assert_xpath (similar to assert_select).

class FunctionalTest(Test):
  def get(self, *args, **kw):
    self.response = app.get(*args, **kw)

  def post(self, *args, **kw):
    self.response = app.post(*args, **kw)

  def assert_response(self, expected):
    pattern = str(expected) if re.search(r'^\d+$', expected) \
                            else (r'^\d+ %s' % expected)
    assert re.search(pattern, self.response.status, re.IGNORECASE), \
           'Response status was not "%s": %s' % (expected, self.response.status)

  def assert_xpath(self, path, expected):
    element = ElementTree.fromstring(self.response.body)
    found_nodes = element.findall('.' + path)
    if type(expected) is int:
      assert_equal(expected, len(found_nodes))
    elif type(expected) is str or type(expected) is unicode:
      assert (True in [(node.text == expected) for node in found_nodes])
    else:
      raise Exception, "Unknown expected value: %r" % type(expected)

If you are doing lots of ListElement equality searches, definitely learn the --tests=foo syntax because testing for matching elements within a list is very slow.

Sometimes I like to load the Rails console against my fixture data to see what's going on in the test environment (i.e. script/console test). To do something similar with GAE, run dev_appserver.py with the parameter --datastore_path="$TMPDIR/nosegae.datastore" (or possibly substitute /tmp for $TMPDIR.

Answer 4

You won't always find one to one equivalents of Ruby testing tools in Python, but there are some great testing tools in Python. Some of the tools that I've found useful include:

• unittest - the xUnit tool included in the Python standard library. It includes all the basics for unit testing.
• doctest - an awesome part of the standard library, it allows you to write tests in the docstrings of functions, classes, modules, methods. It is great at conveying intended API usage. Ian Bicking suggests using doctest for Behavior Driven Development. Doctest fits very well into the Sphinx documentation system (you can make sure all the examples in your documentation pass every time you build the docs).
• nose and py.test are seen as the next-gen versions of unittest. They can run all existing unittest cases, but allow for easier, non-class based unit tests. py.test also allows for distributed execution.
• mock is a nice library for mocking behavior.
• tdaemon watches a directory for updates to your code and will re-execute your test suite. (my personal branch contains a few unmerged improvements).
• Buildbot, Bitten, and even Hudson all work well as full-fledged continuous integration servers for Python code.
• coverage.py computes the code coverage of your code.
• pylint will provide a lint-like analysis of your code, making sure it follows common coding conventions and does not have any common bugs. There is also a "lighter" analysis tool, PyFlakes.
• There are a number of HTTP / Browser testing tools that work well in Python, including Twill, Selenium, and Windmill.

If you are using Django on App Engine, it includes several extensions to unittest that allow you simulate an HTTP client and database persistence.

There are a ton of other tools that I have not used (such as PySpec and Behaviour) that could also be helpful. I haven't seen any mutation testing tool in Python, but I bet there is one out there (I would love to learn what it is).

Happy testing!

Answer 5

Haven't used App Engine, but my feeling for the most popular python testing tools is

• unittest/doctest are the testing packages from the Python standard library. unittest is the xUnit for python.
• nose is a test runner/finder. It has many options, including --with-coverage, which uses coverage to give you code coverage reports.
• pylint is the most featureful lint-checker for python. Useful beyond a syntax checker since it advises on unused variables/functions, when methods should be functions, and more.
• pester (mutation testing)
• buildbot (continuous integration)

You'll probably want to reference this (not quite complete) list of Python Testing Tools.

For BDD, the field was thin last time I checked. Many of the true BDD tools were not usable with nose and/or too limiting in the syntax they required. You might have some luck with spec, which is a BDD-like nose plugin. Just found pyccuracy, which looks a lot like cucumber, but I haven't tried it.

For what its worth, I now just use nosetests -v (the nose runner with --verbose), which will use the first line of the docstring in the test runner output. That is, given a test like:

class TestFoo(unittest.TestCase):
    def testAnyNameHere(self):
        """ Foo should be bar"""
        foo = "bar"
        self.assertEqual(foo, 'bar')

nosetests will give:

$ nosetests -v
Foo should be bar... ok

-----------------------------
Ran 1 tests in 0.002s
OK