Test velocity is a vanity metric. Learning velocity isn't

Somewhere in the last decade, "test velocity" became the KPI of experimentation programmes. Big tech companies run tens of thousands of experiments a year, the reasoning goes, so the path to their results is to run more tests. Agencies now pitch on tests-per-month the way gyms sell sessions.

The number is easy to count, easy to report, and almost entirely disconnected from value.

What the velocity argument gets right

Let's be fair to it first. If win rates are low and unpredictable — and they are — then more attempts means more winners, in the same way more lottery tickets means more prizes. Teams that test rarely also tend to over-invest in each test, inflating stakes and encouraging exactly the peeking and result-laundering that ruins programmes. Some cadence pressure is healthy.

But the argument smuggles in an assumption: that tests are independent draws from the same barrel. They're not — or at least, they shouldn't be.

The quality problems velocity creates

Push a team to triple their test count with the same resources and watch what gets cut. Discovery research goes first — no time for the user interviews and session analysis that produce good hypotheses, so the backlog fills with guesses. Test designs shrink to what's quick to build, which means small tweaks with small effects, which means tests that are underpowered for the traffic available, which means a growing pile of "inconclusive". Analysis gets rushed, losers don't get written up, and the same failed idea gets rebuilt eighteen months later because nobody documented the first attempt.

You end up with the left side of the chart above: forty shallow tests, each independent, each teaching nothing beyond its own outcome. Activity, measured beautifully. Learning, nil.

What compounding actually looks like

The programmes that genuinely compound run sequences. Each test is designed so that its result — win or lose — narrows the next question:

1. A flat redesign test, segmented as pre-registered, hints that price presentation is the sticking point →
2. A test surfacing total cost up front wins with new visitors →
3. A follow-up finds the effect concentrates at the comparison stage →
4. Which motivates a pricing-comparison feature that becomes the biggest winner of the year.

Test 4 was unreachable from a standing start. It only exists because tests 1–3 built the map. That chain of twelve-a-year tests will beat forty disconnected ones on revenue every time, because knowledge — unlike activity — carries forward.

This, incidentally, is the real function of writing up losers (our Graveyard exists for exactly this): a documented loss is a node in the map. An undocumented one is just spent traffic.

The metric to use instead

If you must have a number, track decisions changed per quarter: how many times did a test result alter what the business shipped, priced, wrote or prioritised? It's harder to count, which is precisely the point — it forces the conversation back to what the programme is for. A test that concluded and changed nothing scores zero, however fast it ran.

Velocity has a place: it's a constraint to satisfy (enough throughput that single results don't become political), not an objective to maximise. Optimise for the map, and run exactly as fast as your traffic — and your thinking — can honestly support.