Shadow deployment for test in production

Question

As well known, the most important parts of Test In Production (TiP) are:

• Leverage real customer behavior/data
• Mitigate the customer risk.

I found the Shadow Deployment a promising approach.

With this approach, the V-next product is deployed in production side by side with the V-current. When customer request arrives, a clone of the request is routed to V-next for testing purpose. But the result is never returned to customer. Thus NO risk is exposed to customer. Once the V-next is ready, we can simply switch to it and tear down the V-current.

So it seems the critical parts are:

• How to implement the routing logic for various kinds of projects, such as Web service, Web site, etc.
• How to isolate the data generated during the test because it could have side effect for upstream and downstream systems in production.

My questions are:

• Is there anything else we need to care?
• And is there any best practice/tools/design/solution to address these concerns?

For the specific technology details, please check here: https://stackoverflow.com/questions/14599167/how-to-clone-iis-request-and-redirect-to-a-test-version-site

(Unfortunately, SO community deleted above post...)

Thanks!

Answer 1

We did this at Twitter using Diffy. Not only do you want to capture V-next, you also want to compare it V-current. Further, you want to aggreagate and denoise this comparison over thousands of samples to gain confidence.

Answer 2

A different approach for testing in production is to test on a low percentage of users or to open a new feature only to your QA team on production (dark launches).

There are many tools that can help you test in production, even on real users. Configz.io is a great tool for that.