Reputation: 63
Variation 1 and Variation 5 (see below) both have lower conversion rates than the original yet they are both labeled as more likely than not to outperform the original.
Am I seeing an error? If not, could someone shed some light onto how this Probability of Outperforming Original value is calculated? Thanks.
Original
2,071 Experiment Sessions
1,055 Conversions
50.94% Conversion Rate
0% Compare to Original
0.0% Probability of Outperforming Original
Variation 2
1,028 Experiment Sessions
541 Conversions
52.63% Conversion Rate
3% Compare to Original
69.2% Probability of Outperforming Original
Variation 4
1,786 Experiment Sessions
914 Conversions
51.18% Conversion Rate
0% Compare to Original
61.7% Probability of Outperforming Original
Variation 1
523 Experiment Sessions
258 Conversions
49.33% Conversion Rate
-3% Compare to Original
58.0% Probability of Outperforming Original
Variation 5
837 Experiment Sessions
423 Conversions
50.54% Conversion Rate
-1% Compare to Original
53.2% Probability of Outperforming Original
Variation 3
517 Experiment Sessions
242 Conversions
46.81% Conversion Rate
-8% Compare to Original
44.0% Probability of Outperforming Original
Upvotes: 3
Views: 740
Reputation: 96
It is not any basic or easy calculation to put here some equitation. Calculation of Google Experiments is based on the "problem" of Multi-armed Bandit.
This is a concept that describes any situation in which you want to conduct an experiment in such a way that you maximize your reward
Full description is available on Google Documentation - here: https://support.google.com/analytics/answer/2844870?hl=en
Experiments based on multi-armed bandits are typically much more efficient than "classical" A-B experiments based on statistical-hypothesis testing. They’re just as statistically valid, and in many circumstances they can produce answers far more quickly.
They’re more efficient because they move traffic towards winning variations gradually, instead of forcing you to wait for a "final answer" at the end of an experiment.
They’re faster because samples that would have gone to obviously inferior variations can be assigned to potential winners. The extra data collected on the high-performing variations can help separate the "good" arms from the "best" ones more quickly.
Calculation example is here: https://support.google.com/analytics/answer/2846882
I hope it helps you to understand a bit more how Google is calculating the winner.
Upvotes: 0