abtests.co.uk

Interactive tool

The Peeking Simulator

Every test below is an A/A test: two identical pages, zero real difference. The honest result is "no winner", every time. Now apply the industry's favourite stopping rule — check every day, stop the moment it's significant — and see how often you'd have popped the champagne anyway.

One example A/A test — its p-value wandering day by day. Orange dots are days a daily peeker would have called it "significant".

— of A/A tests "won" under check daily, stop when green. Every one a false positive.
— were significant when read once, at the planned end — as the maths promises, about 5%.
— inflation of your real false-positive rate, caused entirely by the stopping rule.

What you're looking at

Each simulated test flips two identical coins — same conversion rate on both arms — and accumulates visitors day by day. After each day we run a standard two-proportion z-test on the cumulative data, exactly as a dashboard would. If the p-value dips below 0.05 on any day, the daily peeker stops and declares victory. The single-look reader waits until the final day and reads once.

The single-look column sits near 5% because that's what a 0.05 threshold means — for one look. The peeking column is several times higher because every extra look is another chance to catch random noise on a dip. Nothing about the pages differs; the "winners" are manufactured entirely by the stopping rule.

If this is happening to your A/A tests, it's happening to your real tests too — you just can't see it there, because in a real test you can't know the truth. That's the whole trick of this page: with A/A tests, we know the true answer is "nothing", so every green light is visibly a lie.

The fix is unglamorous: fix your sample size in advance and read once, or use sequential methods designed for repeated looks. The full argument is in our essay on peeking.