CODEWITHSUNDEEP
ab-testingabtest
scottystang
scottystang

Reputation: 341

How to A/B test an entire website design

We're building a new website design and instead of cutting over to it 100%, we'd like to ease into it so we can test as we go. The goal would be to have users that visit http://oursite.com to either get the "old" website or the new, and we could control the percentage of who gets the new site by 10%, 50%, etc.

I'm familiar with A/B tests for pages, but not an entire website domain. We're on a LAMP stack so maybe this can be done with Apache VHosts? We have 2 cloud servers running behind a cloud load balancer in production. The new site is entirely contained in an svn branch and the current production site runs out of the svn trunk.

Any recommendations on how I can pull this off?

Thanks you!

Upvotes: 1

Views: 1744

Answers (1)

Lenwood
Lenwood

Reputation: 1404

You absolutely can do this, and it's a great way to quickly identify things that make a big difference in improving conversion rates. It's dependent on a couple of things:

  1. Your site has a common header. You'll be A/B testings CSS files, so this will only work if there's a single CSS call for the entire site (or section of the site).
  2. You're only testing differences in site design. In this scenario all content, forms, calls to action, etc. would be identical between the versions. It is technically possible to run separate tests on other page elements concurrently. I don't recommend this as interpretation of results gets confusing.
  3. The A/B testing platform that you choose supports showing the same version to a visitor throughout their visit. It would be pretty frustrating to see a site's theme change every time they hit another page. Most A/B testing platforms that I've used have an option for this.

Since you're testing substantial differences between versions, I also recommend that you calculate sample sizes before you begin. This will keep you from showing the losing version to too many people, and it will also give you confidence in the results (it's a mistake to let tests run until they reach statistical significance). There are a couple of online calculators that you can use (VisualWebsiteOptimizer, Evan Miller).

Upvotes: 1

Related Questions