The abtests.co.uk blog
Essays on testing properly
Most A/B tests lose. That's not a bug — it's the entire point
Industry win rates hover around one in four or five. If your programme wins more than that, something is probably wrong with your programme, not right with it.
Peeking: how checking your test every morning quietly ruins it
A test checked daily and stopped at the first green light isn't running at 95% confidence. It's often running closer to a coin flip — and the dashboard will never tell you.
The sample size maths nobody does before launching a test
Five minutes with a power calculation kills more bad tests than any amount of post-hoc statistical sophistication. Here's the back-of-envelope version.
The novelty effect: why your winning test faded after four weeks
A variant that wins in week one and evaporates by week six probably didn't stop working. It never worked — it was just new.
Copy testing beats colour testing: an argument from effect sizes
The famous button-colour test is CRO's origin myth, and it has taught a generation of marketers to test the wrong things. Words carry meaning. Hue mostly doesn't.
The Texas sharpshooter problem, or: why your test 'won with iOS users in Leeds'
Slice any flat test into enough segments and one of them will always be a winner. Post-hoc segmentation is where inconclusive tests go to be laundered into success stories.
A/A tests: the cheapest credibility check your programme isn't running
Before you trust a tool with a year of business decisions, make it prove it can tell you nothing. Split identical pages 50/50 and see what happens.
Test velocity is a vanity metric. Learning velocity isn't
The 'ship more tests' movement optimises the number that's easiest to count. What compounds isn't tests run — it's questions answered in a sequence that builds.
CRO for low-traffic sites: what to do when you can't reach significance
Most CRO advice is written for sites with a million sessions a month. Here's the honest playbook for everyone else — and it still involves testing, just not the kind you think.
Your losing tests are an asset. Start treating them like one
Winners get slides; losers get silence. But a properly documented test archive is the only CRO output a competitor can't copy — and the reason we run a public graveyard.