Most A/B tests lose. That's not a bug — it's the entire point

Ask any honest experimentation team and they'll tell you the same thing: most of the tests they run don't win. Across the industry, the numbers that get quoted are remarkably consistent — somewhere between one in four and one in eight tests produces a clear, positive result on the primary metric. The rest are flat or negative.

New clients find this horrifying. "So we're paying to be wrong 75% of the time?"

No. You're paying to find out you're wrong 75% of the time — before you ship the change to everyone.

The counterfactual nobody prices in

Every losing test is a change your team believed in enough to build. Design signed it off. Someone senior probably championed it. Without a test, that change goes live to 100% of traffic and quietly costs you conversions for months, with nobody the wiser because nothing is being measured against a control.

A test that loses hasn't wasted money. It's rescued you from a full rollout of a bad idea, at a fraction of the cost. The 5% dip you saw in the variant ran on half your traffic for three weeks — not all of it, forever.

High win rates are a smell, not a flex

When an agency tells you they win 60–70% of their tests, one of three things is usually true:

1. They're testing the obvious. Fixing broken forms and removing dead links "wins", but you didn't need an experiment to know that — you needed a bug ticket.
2. They're peeking. Stop the test the moment it goes green and you'll harvest a lovely crop of false positives. (We built a simulator so you can watch this happen.)
3. They're only counting the tests they finished. Quietly abandoning losers before they conclude does wonders for a win rate.

A programme winning one test in four, with pre-registered success criteria and full sample sizes, is worth vastly more than one "winning" two in three with none of that discipline.

What a healthy loss looks like

A good losing test comes with a documented hypothesis, a clear read on why it lost, and a next step. "We believed students choose accommodation on price, so we led with weekly rents on listing cards. Conversion fell 6%. Exit surveys suggest distance-to-campus matters more at this stage." That's not a failure — that's a finding, and it just redirected your whole roadmap.

This is why we keep a public Test Graveyard. Losers, written up properly, stop you re-running the same bad idea in eighteen months when the team has turned over, and they compound into something no competitor can copy: an evidence base about your customers.

The maths of the whole programme

Suppose you run 20 tests a year. Five win at an average of +6% on the metric they target; the losers cost you nothing at full rollout because you never rolled them out. The programme's return is the compound of the winners minus the (bounded, temporary) cost of running the losers. That's a good trade in almost any traffic scenario — provided the wins are real, which brings us back to discipline.

Chase the win rate and you'll get a worse programme. Chase the truth and the win rate takes care of itself.