CODEWITHSUNDEEP
pythontestingmockingbotoamazon-dynamodb
David Eyk
David Eyk

Reputation: 12591

How do I test a module that depends on boto and an Amazon AWS service?

I'm writing a very small Python ORM around boto.dynamodb.layer2. I would like to write tests for it, but I don't want the tests to actually communicate with AWS, as this would require complicated setup, credentials, network access, etc.

Since I plan to open source the module, including credentials in the source seems like a bad idea since I will get charged for usage, and including credentials in the environment is a pain.

Coupling my tests to the network seems like a bad idea, as it makes the tests run slower, or may cause tests to fail due to network errors or throttling. My goal is not to test boto's DynamoDB interface, or AWS. I just want to test my own code.

I plan to use unittest2 to write the tests and mock to mock out the parts of boto that hit the network, but I've never done this before, so my question boils down to these:

  1. Am I going about this the right way?
  2. Has anyone else done this?
  3. Are there any particular points in the boto.dynamodb interface that would be best to mock out?

Upvotes: 14

Views: 7967

Answers (3)

David Eyk
David Eyk

Reputation: 12591

To answer my questions fully:

  1. As with @pcalcao's answer, mocking out the service was the right thing to do. It wasn't even as hard as I thought it would be--the test code is only marginally longer than the code under test, and most of it is tests, not plumbing.

  2. Thanks to @gamaat for getting me to take a second look, boto does do this in its own tests, mocking at the actual transport interface level, in httplib.

  3. Mocking the boto.dynamodb.layer1 interface (along with boto.connect_dynamodb) turned out to be the right thing to do. Setting spies on boto.dynamodb.layer2 and boto.dynamodb.table also helped. Once I began to understand it, I found that mock is a joy to work with.

Here's my solution, BSD licensed. I'll be posting the entire library to PyPI once I've dog-fooded it a bit longer and put together some proper documentation. I've posted it to PyPI.

Upvotes: 4

garnaat
garnaat

Reputation: 45906

All tests in boto were originally integration tests that hit live service endpoints. We still have those tests but have also started adding mocked unit tests. You might want to check out what's there so far for examples.

Upvotes: 3

pcalcao
pcalcao

Reputation: 15990

I think you have the right approach, you definitely don't want your tests tied up with actual communication with AWS. Mocking the service is definitely the right thing to do here.

Upvotes: 4

Related Questions