Statistics · 8 June 2026
Peeking: how checking your test every morning quietly ruins it
A test checked daily and stopped at the first green light isn't running at 95% confidence. It's often running closer to a coin flip — and the dashboard will never tell you.
Here's a habit almost every team has, and almost no tool warns you about: opening the testing dashboard each morning, glancing at the significance figure, and stopping the test the day it turns green.
It feels diligent. It's actually the single most common way good tests produce bad answers.
Why peeking breaks the maths
A p-value threshold of 0.05 means this: if there is genuinely no difference between the variants, a test run to its planned end has a 5% chance of falsely showing one. That guarantee holds for one look, at the end.
Check daily and you're taking dozens of looks. The p-value of a null test doesn't sit politely above 0.05 — it wanders. It dips, it spikes, it flirts with the threshold. Each peek is another chance to catch it on a random dip, and if your rule is "stop when green", you will catch it. Simulations of A/A tests (identical pages, no real difference) checked daily over four weeks produce a "significant winner" 25–40% of the time, depending on traffic. Your 95% confidence is, in practice, closer to 60–75%.
Don't take our word for it — we built the Peeking Simulator so you can run a thousand A/A tests yourself and watch the false positives stack up.
"But the tool said significant"
It did — at that instant, using maths that assumes you weren't going to look until the end. The tool doesn't know your stopping rule. Statistical significance is a property of a procedure, not a snapshot, and "stop at first green" is a different, much weaker procedure than "run to N, then read".
This is also why so many "winners" evaporate on rollout. The uplift was never real; it was a peek harvested at the top of a random wobble.
What to do instead
Fix the sample size in advance. Decide the minimum effect worth detecting, calculate the sample you need, and don't read the result until you get there. Boring, unfashionable, correct.
If you must monitor, monitor for harm only. It's legitimate to watch a test for guardrail breaches — a variant tanking revenue 20% shouldn't run for three more weeks. Set an explicit, extreme threshold for early stopping for harm, and treat any early stop as inconclusive for the win.
Or use methods built for looking. Sequential testing and Bayesian approaches with proper stopping rules exist precisely because humans can't stop peeking. They cost you some sensitivity in exchange for honesty. That's usually a fair trade. What isn't fair is using fixed-horizon statistics with a sequential stopping habit — the worst of both worlds, and the industry default.
The cultural fix matters more than the statistical one
Teams peek because stakeholders ask "how's the test doing?" in Monday standups. The real fix is upstream: when a test launches, publish its end date and planned sample alongside the hypothesis, and make "we'll know on the 24th" an acceptable answer. A test in flight has no interim result worth reporting — only a countdown.
The dashboard will still be there every morning, glowing green three days in. Let it glow.