A/B Testing - How it Works

You can monitor the visitor’s activities using statistics and analysis to decide the version that yields a higher conversion rate. A/B Testing results are normally given in fancy mathematical and statistical terms, however the meaning behind the numbers are actually quite straightforward. There are two important methods through which you can check conversion rates using A/B Testing −

  • Sampling of Data
  • Confidence Intervals

Let us discuss these two above methods in detail.

Sampling of Data

The number of samples depend upon the number of tests performed. The count of conversion rate is known as a sample and the process of collecting these samples is called as sampling.


Let us say you have two products A and B, you need to collect sample data as per its demand in the market. You can request a few people to choose from product A and B and then request them to participate in a survey. As the number of participant’s increase, it will begin showing a realistic conversion rate.

There are many different tools that can be used to determine the correct number of sample size. One such free tool available is −



Confidence Intervals in A/B Testing

Confidence interval is the measurement of deviation from the normal on the multiple number of samples. Let us accept that 22% of people prefer product A in the above example, with ±2% of confidence interval. This interval indicates the upper and lower limit of the individuals, who opt for Product A and is also called as margin of error. For best outcomes in this average survey, the margin of error should be as small as possible.


Let us assume that in Product B, we have included a minor change and then performed A/B Testing on these two products. Confidence interval product A and B are 10% with ±1% and 20% with ±2% respectively. So this shows that a minor change has increased the conversion rate. If we ignore the margin of error, conversion rate for test variation A is 10% and conversion rate for test variation B is 20%, for example a 10% increase in the test variation.

Now, if we divide the difference by control variation rate 10% ÷ 10% = 1.0 = 100%, it shows an improvement of 100%. Hence, we can say that A/B Testing is a technique based on mathematical strategies and analysis. There are various online tools that can be utilized to calculate A/B significance.



Input your Topic Name and press Enter.