CODEWITHSUNDEEP
pythongoogle-app-engine
Richard Corfield
Richard Corfield

Reputation: 2529

InvalidModuleError() when using Testbed to Unit Test Google App Engine

I’ve been wrestling with a couple of errors when trying to write some Python unit tests for a Google App Engine app that consists of a number of modules.

I have been following the guidance given on https://cloud.google.com/appengine/docs/python/tools/localunittesting

To begin with I was getting the following:

ERROR:root:AssertionError('No api proxy found for service "modules"’,)

However I determined that this was because I had not initialised Testbed correctly and needed a separate call to:

self.testbed.init_modules_stub()

This is despite already calling:

self.testbed.init_all_stubs()

This seems odd to me, but is not the main problem ... Now I've moved past that error, and instead I'm getting:

ERROR:root:InvalidModuleError()

The code is quite straightforward. Here's the relevant parts of the Test case:

def setUp(self):
    self.testbed = testbed.Testbed()
    self.testbed.activate()
    self.testbed.init_all_stubs
    self.testbed.init_modules_stub()

def test_should_submit_a_task(self):
    post_content = '{ "bucket": "/test/", "filename", "test", "operation": "read" }'
    request = webapp2.Request.blank('/path/to/module/method', POST=post_content)
    response = request.get_response(main.application)

    self.assertEquals(response.status_int, 200)

The line from the code under test where the error is thrown is as follows:

     host = get_hostname(queue)

I can see that 'queue' has been correctly initialised with the name of the module.

The comments in google_appengine from get_hostname() state:

"Raises: InvalidModuleError if the given moduleversion is invalid."

So for some reason my moduleversion is invalid.

So, is it it necessary to manually pass a module version to get_hostname() when the code is under test?

Or have I failed to initialise the testbed in some way to ensure that the module's version is valid?

EDIT: I've continued to work on this and have traced through the Google App Engine code to the class _LocalFakeDispatcher in request_info.py . That class sets up some defaults for use in testing. The problem is, my test arrives here trying to determine if the module is valid, but it doesn't match one of the defaults in this stub, and so an Invalid Module Error is eventually returned.

Is there some way I can override the defaults in this dispatcher to set it up populated with the expected module names and versions?

See: 

class _LocalFakeDispatcher(Dispatcher):
  """A fake Dispatcher implementation usable by tests."""

  def __init__(self,
               module_names=None,
               module_name_to_versions=None,
               module_name_to_default_versions=None,
               module_name_to_version_to_hostname=None):
    super(_LocalFakeDispatcher, self).__init__()
    if module_names is None:
      module_names = ['default']
    if module_name_to_versions is None:
      module_name_to_versions = {'default': ['1']}
 etc. ...

Many thanks

R.

$gcloud --version Google Cloud SDK 0.9.44

app 2015.01.15 app-engine-go-darwin-x86_64 1.9.17 app-engine-java 1.9.17 app-engine-managed-vms 2014.11.03 app-engine-python 1.9.17 etc.

Upvotes: 3

Views: 731

Answers (1)

Alex Martelli
Alex Martelli

Reputation: 882841

The issue with needing to call init_modules_stub is connected to what release of the SDK (and thus of testbed/__init__.py) you have; that's been added to init_all_stubs in the current release (not sure when exactly) so an upgrade should let you remove the need for the explicit call. But as you say, not the main problem.

But on to the more substantial problem -- by my lights you've done nothing wrong, because no documentation says you should do anything in particular to initialize the modules' stub.

Fortunately, a work-around isn't too terrible. Specifically, you could have, early in your unit-test code, the initialization:

from google.appengine.api import request_info

# edit all_versions per modules & versions thereof needing tests
all_versions = {'default':[1], 'andsome':[2], 'others':[1]}
def_versions = {m:all_versions[m][0] for m in all_versions}
m2h = {m:{def_versions[m]:'localhost:8080'} for m in def_versions}

request_info._local_dispatcher = request_info._LocalFakeDispatcher(
    module_names = list(all_versions),
    module_name_to_versions = all_versions,
    module_name_to_default_versions = def_versions,
    module_name_to_version_to_hostname = m2h)

assuming of course that these are the module names and versions you want!

Yes, it should definitely be easier (the testbed or some module should expose a function doing this -- ideally by parsing appropriate yaml files, but at least with explicit arguments) and, very importantly, it should be well documented.

Having been the first author of a precursor of testbed's first version five years ago, I personally apologize for not keeping an eye on it (sorry -- I was busy with very different jobs in the meantime! -- but, being a fanatic about unit-testing, I should have used some 20% time on this).

Please do open a feature request about exposing this nicely and documenting it, and thanks both for your patience and for your excellent "detective work" identifying the crux of the problem!

Upvotes: 4

Related Questions