CODEWITHSUNDEEP
google-app-enginegoogle-website-optimizermultivariate-testing
Danger
Danger

Reputation: 2113

How do you use GAE traffic splitting for multivariate testing?

The new GAE traffic splitting feature was released today and one of its intended uses is for multivariate testing. This would seem to imply that you must build and deploy multiple versions of your application, one for each variant of the test, which implies that you've branched your version control system (one branch per test) as well. This seems like an inefficient way to manage testing. It also does not appear to include any kind of framework for managing the tests, unlike Google's Website Optimizer which includes user interfaces for managing multiple tests and also includes all the statistics gathering and analysis tools.

Is anyone thinking of using the new traffic splitting feature for multivariate testing? Why use this over Website Optimizer? How would you set up traffic splitting tests - would you branch your version control system and also write your own infrastructure for measuring and and analyzing tests? Or rather than write your own code, is there a multivariate testing library that would work well for this purpose?

Upvotes: 1

Views: 533

Answers (1)

matt burns
matt burns

Reputation: 25370

You choose the traffic splits between versions of your deployed application, not necessarily different revisions in your version control. The code in each version doesn't have to be very different, if at all. In java you define the version in appengine-web.xml.

Say for example you wanted to run a simple A-B test to compare a version of your site with ads, against one without. You may be looking for effects on bounce rate, load time, ad effectiveness, whatever. In your code, you could just wrap the ad displaying code in a block of code with a flag like:

if (ADS_ON) {
   // display ad...
}

You set the application version to "ads-on" and set ADS_ON to true and deploy. You would then set the application version to "ads-off" and set ADS_ON to false and redeploy.

Now that I've typed it out, yeah, I agree it's a bit clunky, but this new feature still allows you to do things your weren't able to do before. You will be able to look at the appengine dashboard for each version and compare how they consume resources.

If you wanted to test the running costs of two completely different implementations then it would make sense to have two branches in your version control.

Upvotes: 1

Related Questions