The A/B Test You're Running Is Wrong: A Guide to Statistical Power

Most A/B tests are stopped too early, run with too little traffic, or declare winning variants based on data that cannot support the conclusion. Statistical power — the probability that your test will detect a real effect if one exists — is the variable almost nobody calculates before running. The result is an industry built on false positives.

The problem is that conversion rate data is noisy, especially early in a test. In the first few days, random fluctuations will frequently produce apparent leads for one variant or the other. If you are watching your test and stopping it when the results look favorable, you are guaranteed to overestimate your win rate. Simulations of this stopping behavior — where a test is monitored daily and stopped when p < 0.05 is reached — show false positive rates of 25 to 40 percent, compared to the 5 percent the test was designed to produce. You are declaring winners at six to eight times the appropriate rate.

This is called optional stopping or peeking bias. The mathematical reason is that your significance threshold was calculated assuming you would look at the data exactly once — at the end of a pre-specified sample size. Looking at data multiple times and stopping when it looks significant inflates your Type I error rate in proportion to the number of looks. Running a test until it reaches significance is not a completion criterion. It is a method for generating false positives.

The implication that is almost never stated: at 80% power, if your variant has a real effect, you will miss it 20% of the time. Those missed effects are called Type II errors, or false negatives. A test that fails to reach significance is not proof that the variant did not work. It is entirely consistent with the variant working but your test being too small to detect it.

Power is a function of three variables: your significance threshold (usually α = 0.05), the baseline conversion rate of your control, and the Minimum Detectable Effect (MDE) — the smallest improvement you care to detect. If you want to detect a 5% relative improvement on a 2% conversion rate, you need far more traffic than if you are only trying to detect a 20% improvement. The relationship between these variables is not intuitive. At 80% power, detecting a 5% relative improvement on a 2% conversion rate requires approximately 80,000 visitors per variant. Detecting a 20% relative improvement on the same rate requires approximately 5,000.

A reasonable heuristic for DTC conversion optimization: if the implementation cost of a change is low — a headline rewrite, a button color — an MDE of 5 to 10% relative improvement is sensible. If the implementation cost is high — a checkout flow rebuild, a new landing page architecture — you need to detect larger effects to justify the resource investment. Setting a higher MDE means you need less traffic and shorter run time. Setting a low MDE means you need more traffic and more time, which has real costs of its own.

Most A/B testing platforms include sample size calculators. The output is a minimum number of visitors per variant needed before the test can be read. Treat this as a hard floor, not a guideline. The test cannot be read — even if the numbers look significant — until this minimum has been reached. If you reach your sample size before reaching significance, the correct conclusion is that your variant likely does not produce an improvement above your MDE. That is a valid and actionable result.

The right question when setting an MDE is: what is the smallest lift that would change a business decision? If you are testing a checkout page change and your current checkout conversion rate is 3.2%, a 5% relative improvement would take you to 3.36%. Is that meaningful enough to build a roadmap around? If yes, power your test to detect 5%. If the answer is that you only care about lifts above 15%, then power your test for 15% and accept the risk of missing smaller improvements.

There is no objectively correct MDE. There is only the MDE that is consistent with your cost structure, your decision process, and the traffic you have available to run the test. Choosing it explicitly — before running — is what separates a test you can interpret from one you cannot.

Several platforms — including Optimizely's Stats Engine and VWO's SmartStats — implement Bayesian or sequential testing frameworks that allow continuous monitoring. These are appropriate tools if you understand their tradeoffs. Bayesian testing shifts from asking whether an effect is statistically significant to asking what the probability distribution over the true effect looks like. This framing is often more useful for business decisions but requires specifying a prior belief about the likely effect size, which introduces a subjective element that fixed-horizon frequentist testing avoids.

If the confidence interval on your measured difference is narrow and centered near zero, the evidence is good that the variant is essentially equivalent to control. File the result, update your hypothesis about what works in this context, and move to the next test. If the confidence interval is wide — spanning meaningful improvements and meaningful reductions — the test was underpowered and the result is uninformative. The correct response is to re-run with an adequate sample size, not to conclude the variant failed.

Null results have value that is systematically underweighted in most experimentation programs. A well-powered null result rules out the hypothesis that a change generates a lift above your MDE. That is information. Over time, a program that accurately records null results builds a meaningful map of what does not move the needle in your specific conversion context — which is often more durable and more valuable than a list of positive results that may not replicate.