Statistical and Financial Considerations in Website Optimization
What's a high-confidence winner?
A high-confidence winner is a tested combination that you can feel confident will bring you more conversions than your original website configuration. Here's an example: Suppose you tested three combinations: X, Y, and Z. X showed a 23% chance of beating the original, Y showed a 44% chance of beating the original, and Z showed a 99% chance of beating the original. In that case, Z (with its very high probability of improvement) would be marked as a high- confidence winner.
GWO will alert you in your report whenever a high- confidence winner is found in one of your experiments. You'll see the phrase 'High-confidence winners found,' followed by details on those high-confidence combinations. There may be zero, one, or more than one high-confidence winner in any report, depending on the results of your experiments.
You may not see a high-confidence winner with every experiment. In some cases no significant difference will be found between the combinations tested and your original. In that case, no high-confidence winner will be declared.
Sometimes you simply need more data to be able to reach a level of high confidence. A tested combination typically needs around 200 conversions for you to judge its performance with certainty.
If the results are inconclusive, what do I do?
If the results are inconclusive, it may mean that you weren't able to sample a significant number of users, or that there wasn't a significant difference between the combinations tested. If there wasn't a significant difference between combinations, but several combinations performed better than your original content, you or your webmaster may wish to replace the original content on your live site with the HTML content for one of the better- performing combinations. Alternatively, you may prefer to edit and run your experiment again.
Conversions/impressions is just that -- the raw data of how many conversions and visits a particular combination generated. It represents the number of visitors who reached the conversion page after viewing the test page where the given combination was presented.
Let's take a look, in greater detail, at the Original, Combination 11 and/or Combination 7 rows of in the Combinations section shown in Figure 6.Estimated Conversion Rate.
- 125/401 => 31.2% (point average for Original)
- 160/411 => 38.9% (point average for Combination 11)
- 143/453 = > 31.6% (point average for Combination 7)
Estimated Conversion Rate Range
Figure 8. Standard error of sample proportion (Eqn. 1) and confidence interval for a proportion (Eqn. 2)
The Website Optimizer uses an 80% confidence level to estimate conversion rate range; that is, the real conversion rate delivered by a combination will fall within the estimated range 80% of the time. The Estimated Conversion Rate Ranges reported in Figure 7 use Equation 2 with a z- multiple or 1.28 (signifying 80%).
Chance to Beat Original
Using the formula for confidence interval for the difference between proportions in Equation 4, the Excel spreadsheet shown in Figure 8 computes the values shown in the Chance to Beat Original column in the report shown in Figure 7.
Figure 9. Standard error of difference between sample proportions (Eqn. 3) and confidence interval for difference between proportions (Eqn. 4)
Click here for larger image
Figure 10. Using Microsoft Excel for the calculations in Equations 3 and 4.
- Combination 11's improvement over Original 38.9 / 31.2 => 24.7%
- Combination 7's improvement over Original 31.6 / 31.2 => 1.28%
So, should we replace the Original landing page with either the one that has Combination 11 or Combination 7? Not necessarily.
Suppose, for example, that the high conversion rate for Combination 11 came from the fact that it displayed a $350 price for a camera and the only slightly higher conversion rate for Combination 7 came from the fact that it displayed a $425 price for the same camera, while the Original page offered the camera for $450. And, suppose that your cost to make and advertise this camera was $300.
Then, for every 100 customers who arrived on your landing page, you could expect to make a profit of about
- $50(.3893)(100) = $1946 for Combination 11
- $125(.3157)(100) = $3946 for Combination 7
- $150(.3117)(100) = $4674 for Original
This is a pretty extreme example, but it does demonstrate the point that achieving your business goal - maximum profit, in this case - isn't necessarily the same as maximizing your conversion rate. I'll discuss the financial aspects of Website optimization further in my next article, Statistical and Financial Considerations in Website Optimization: Part 2.
Page 3 of 5