How do you do automated testing on Google App Engine?

Question

I am creating a app on Google app engine and am wondering if there are ways to do automated testing with python.

Thanks!

Answer 1

We are generally not testing too much. We once had a "80% test coverage" rule but found this doesn't make us better or faster. Most code and data structures we use are designed quite defensively so there is seldom harm which can't be undone. Our users prefer fast turnaround times to 100% uptime.

We have two apps setup: my app.appspot.com and my app-test.appspot.com. The whole codebase is designer to ensure app-test.appspot.com never changes state in external systems.

occasionally we copy the data from app.appspot.com to app-test.appspot.com. It can get messy, because id generation counters for the datastore don't get updated but it works good enough.

We develop on both systems. Frontend development is done mostly on app.appspot.com and experiments with the backend are done on app-test.appspot.com.

We have three branches: master, rc and production.rc gets updated from master and production from rc. rc is deployed daily to rc.app.appspot.com by or operations them. production is deployed weekly to production.app.appspot.com (which is also reachable via an other app name.

Developers usually deply to dev-whoami.app.appspot.com for experimenting. We use the development server very little because wee need a lot of data from the datastore.

Now to testing: we mostly use acceptance tests. We have a little framework called resttest_dsl which we use to describe tests like this:

client.GET('/').responds_access_denied()
client.GET('/', auth='user').responds_html()
client.GET('/admin').responds_access_denied()
client.GET('/admin', auth='user').responds_access_denied()
client.GET('/admin', auth='admin').responds_html()
client.GET('/artikel/').responds_with_html_to_valid_auth()
client.GET('/artikel/?q=Ratzfratz', auth='user').responds_html()
client.GET('/api/ic/v3/a/14600/03/zukunft.json').responds_with_json_to_valid_auth()
client.GET('/kunden/SC50313/o/SO1194829/', auth='user').responds_html()
client.GET('/api/masterdata/artikel/v2/artnr/14600/bild=s210').redirects_to('...')

hostname and credentials have defaults but can be overwritten by environment variables. Most errors we ever have fixed have a regression test in there. We use Makefiles to drive the whole stuff. Eg.g:

deploy:
appcfg.py update -V dev-`whoami` -A app .
TESTHOST=dev-`whoami`.app.appspot.com make resttest
open http://dev-`whoami`.app.appspot.com/

Deployment always happens from the central git repository like this:

deploy_production:
rm -Rf tmp
mkdir tmp
(cd tmp ; git clone git@github.com:user/app.git)
(cd tmp/app ; git checkout production ; make dependencies)
(cd tmp/app ; git show-ref --hash=7 refs/remotes/origin/production > version.txt)
appcfg.py update -V "v`cat tmp/app/version.txt`" -A app tmp/app
(cd tmp/huWaWi ; TESTHOST="v`cat version.txt`".app.appspot.com make resttest)
appcfg.py update -V production -A app tmp/app
appcfg.py backends -V production -A app tmp/huWaWi app

We first deploy to a version tagged with the current revision on AppEngine. We then run resttest.py against this freshly deployed version. On failure the mmake stops execution. If no failure occurred the "production version" is deployed.

We also run mandantory pep8, pyflakes and pylint checks on source code checkin.

All in all we have very simple minded tests but run them a lot and against production code and data. For us this catches most of error we make which relatively little effort.

Answer 2

David Robinson refers to the development unit testing. If you are looking for automated user(production) testing using python, go for selenium rc or selenium webdriver(improved version & standalone).

You can do wonders with selenium RC.

Refer to http://seleniumhq.org/projects/webdriver/

Answer 3

I use gaeunit - http://code.google.com/p/gaeunit/ - which may or may not suit your needs but once its going it's pretty easy to add to. I also added an xml output so that I can stuff the results back into a junit analyser so my jenkins can report back after code checkins that nothing broke.