In this section, we will be discussing on how to measure the results and simplify statistics of the testing done on the site.
Researchers may or may not understand statistics. But, A/B Testing tool proves to be a savior by simplifying those statistics. So a lot of calculation can be avoided. Most of the testing tools are consistent in utilizing a 95% criteria as a successful goal completion.
That means out of 10 you are a winner by 9. Let us take an example. Your testing tool report is as per the following −
This report predicts a more or less .20% conversion rate variance at the 95% interval. Statistically, the objective range is between 1.76 and 2.06.
While planning a test I keep two objectives in mind. First, is boosting revenue and the other is getting insights about what prompted a higher ROI.
For instance, in a case study we get diverting traffic to product page rather than category page or homepage improves conversion rates or not. We took three variations, in one we coordinated traffic on the homepage loaded with categories and subcategories directing further to the product page. In the second, we directed traffic to category page adding filters. In the third, we directed it directly on the product detail page with buy button.
To my surprise, the third variation won. This is just the information required by a buyer about the product. This lets us figure how conversion rate lift and continuous improvements can make us develop our leads.
Undoubtedly, including many variations and insights over tests gave us the website redesign.
Let me clear it out. NOT ALL TESTS WIN. Yes, it is painful yet true.
There are tests that gives you results in flying colors. There are others that after so many tries too would be without an outcome. But if you plan a test with an insight driving segmentation, you can have a new hypothesis to be tested on. Not all test helps you to improve revenue.
Take an example to understand. There are three campaigns with various conversion rates.
Anyone will blindly say ‘Campaign B’ is a super performer. But let us dig some more.
See closely, ‘Campaign B’ is too small to be statistically significant. Campaign B with 1 transaction with one visit will give 100 percent conversion rate. ‘Campaign A’ performs over ‘Campaign C’. While concluding results, there are several factors that need to be looked at and it may differ every time. It is you who need to look at all insights and decide the outcomes.