The sample size maths nobody does before launching a test

Most doomed tests are doomed before launch, and for the same unglamorous reason: not enough traffic to detect the effect being hunted. The test runs for three weeks, wobbles around zero, gets called "inconclusive", and everyone quietly resents the testing programme a little more.

The fix takes five minutes and happens before you build anything.

The only three numbers that matter

To know roughly how many visitors a test needs per variant, you need:

1. Baseline conversion rate — what the page converts at today. Say 3%.
2. Minimum detectable effect (MDE) — the smallest uplift you'd actually care about. Be honest: is a 2% relative lift worth the build cost? Usually you're hoping for 5–10%.
3. Your appetite for being wrong — conventionally 95% confidence and 80% power, and you can leave those alone.

A serviceable back-of-envelope formula for visitors per variant is:

n ≈ 16 × p × (1 − p) / (p × MDE)²

where p is the baseline rate and MDE is relative. For a 3% baseline and a 5% relative lift: n ≈ 16 × 0.03 × 0.97 / (0.0015)² ≈ 207,000 per variant. For a 10% lift, about 52,000 per variant.

Read those numbers again, because they're the ones that quietly kill most testing roadmaps. A site doing 40,000 sessions a month to the tested page cannot detect a 5% lift this quarter. It just can't, and no dashboard setting changes that.

What the calculation actually buys you

Not precision — these are estimates, and that's fine. What it buys you is triage:

• Green: the test reaches sample in 2–4 weeks. Build it.
• Amber: it needs 6–8 weeks. Only worth it for a big, strategic question — and you must commit to the full duration up front, novelty effects and all.
• Red: it needs six months. Don't run it. Test something bolder (bigger expected effect = smaller required sample), test further up the funnel where traffic is higher, or make the change on judgement and measure it another way.

That last option scandalises people, but it shouldn't. A/B testing is one instrument, not a religion. Shipping a well-reasoned change without a test is a better decision than running a test that cannot conclude — because the fake test gives you fake confidence in whatever noise it produces.

The MDE conversation is really a business conversation

Notice that the sample size explodes as the MDE shrinks. That's not a statistical inconvenience; it's the maths telling you something true: small effects are expensive to prove. When a stakeholder wants to test a tweak — a slightly different icon, a reordered footer — the power calculation is how you make the cost visible. "We can test that, and it will occupy our entire test slot for four months" tends to refocus minds on changes big enough to matter.

This is also the honest answer to why we push clients towards fewer, braver tests. It isn't a creative preference. It's arithmetic.

Do it every time

Before any test launches, the one-line ritual: baseline, MDE, required sample, expected duration, written at the top of the test doc. If the duration is unacceptable, the test doesn't launch. Five minutes, and your programme stops accumulating inconclusive wreckage.