CODEWITHSUNDEEP
optimizely
Haim
Haim

Reputation: 1167

Optimizely reliability

We started using Optimizely and we're blown by the results so we started creating aggressive campaigns. But then we realized something was wrong and I would appreciate any help we can get.

We tested (for example) on the homepage instead of a blue CTA, we created a variation with a green CTA and we saw a 35% increase in conversions with the variation. So then we switched things and set the green CTA as the control and the blue CTA as the variation. If it was really true that the green CTA is better we should now be seeing a 35% decrease in conversions with the variation of the blue CTA. Instead, the variation came back with a higher conversion rate.

I can't pinpoint what we're doing wrong, but in general all the variations come in with a higher success rate than the control, but if we switch things around and set the variation as the control, the results don't switch.

Upvotes: 1

Views: 142

Answers (3)

user6810289
user6810289

Reputation: 31

Are you making these adjustments to an experiment that has already been running? You should keep in mind that users are bucketed once you've started an experiment so making adjustments like switching who belongs to a control or not, after or whilst an experiment has run will not be starting you fresh so to say. If you're making the adjustments you describe then you'll need to create a new experiment.

Upvotes: 0

Julio Farfan
Julio Farfan

Reputation: 1

Try: On Results page apply different segments to identify improvement in each variation I recommend setting a 90% of statistical significance If you tested in 2 separate test, maybe you didnt let it the right time i usually prefer to have them by a week to avoid seasonality,

Upvotes: 0

Zsolt
Zsolt

Reputation: 1

How do you measure conversions? It's usually a good practice to measure in more ways than once - for example, for a CTA experiment I would measure clicks, landing on the page where the link redirects, valid form submissions etc. Expanding your view on the data and see how a change will affect other parts of the user experience is always a good idea.

Upvotes: 0

Related Questions