Sellerbench

A/B test significance calculator

Find out whether the difference between two versions of a page is real or just chance. Enter visitors and conversions for each variation to get the lift, p-value and a clear result. Planning a test? Use the sample size calculator first.

Variation A (control)
Variation B (test)
Test settings

Result
–
p-value
–

How the test works

This calculator uses a two-proportion z-test, the standard method for comparing conversion rates. It asks how likely a gap this large would be if both versions really performed the same. That probability is the p-value. If it is below your threshold (5% for 95% confidence), the result is called statistically significant.

Conversion rate = conversions ÷ visitors
Pooled rate = (conversions A + conversions B) ÷ (visitors A + visitors B)
z-score = (rate B − rate A) ÷ √(pooled rate × (1 − pooled rate) × (1 ÷ visitors A + 1 ÷ visitors B))
p-value (two-sided) = 2 × (1 − Φ(|z|)), where Φ is the normal distribution function

Worked example

Version A converts 500 of 10,000 visitors (5.00%). Version B converts 570 of 10,000 (5.70%), a relative lift of 14%. The z-score is 2.20 and the two-sided p-value is 0.028. That is below 0.05, so B is better at 95% confidence, but not at 99%, where the threshold is 0.01.

Mistakes that make results unreliable

Checking the result every day and stopping the moment it looks significant inflates false positives, so decide the sample size in advance with the sample size calculator and wait for it. Run the test for whole weeks so day-of-week effects balance out. Test one change at a time, and do not trust a result with very few conversions. Significance tells you a difference is probably real, not that it is large or valuable, so also look at the confidence interval and the revenue impact.

To estimate what a lift is worth, use the CRO uplift calculator.

Frequently asked questions

What does statistically significant mean in an A/B test?

It means a difference this large would be unlikely if both versions performed the same. At 95% confidence the p-value must be below 0.05.

What is the difference between one-sided and two-sided tests?

A two-sided test checks for any difference, better or worse. A one-sided test only checks whether B is better than A, so it needs less data but cannot detect that B is worse. Decide before you look at the data.

Can I stop my test as soon as it is significant?

No. Stopping early on a good-looking result raises the chance of a false win. Set the sample size in advance and run the test to the end.

How many conversions do I need?

Enough to reach the sample size from the planning calculator. As a minimum, treat results with fewer than about five conversions in a variation as unreliable.