CRO for low-traffic sites: what to do when you can't reach significance

Run the sample size numbers for a site doing 20,000 sessions a month and the conclusion is blunt: you cannot detect a 5% lift on a 3% conversion rate this year. Most consultancies respond by not mentioning it and selling you the testing programme anyway. Here's what we tell clients instead.

First: shrink the problem, not the ambition

Move the test up the funnel. Your checkout may see 2,000 users a month, but your landing pages see 20,000 and your category pages 12,000. Micro-conversions — enquiry starts, add-to-basket, availability checks — happen at 5–10× the rate of final conversions, and higher baseline rates slash required samples dramatically. Testing "clicked through to booking" instead of "completed booking" can turn a nine-month test into a five-week one. You lose some certainty that the upstream metric translates downstream; you gain a test that actually concludes. Monitor the downstream number as a guardrail.

Test bigger swings. Sample size scales with the inverse square of the effect you're hunting — halve the effect, quadruple the traffic needed. Low-traffic sites therefore cannot afford subtle tests. A reworded button is invisible at your volume; a fundamentally different page proposition might not be. This is the happy irony of low traffic: it forces you toward the brave, meaningful tests high-traffic teams often avoid. One big test run to completion beats five timid ones that never conclude.

Loosen the thresholds — knowingly. The 95%/80% convention comes from academia, where a false claim pollutes the literature forever. A business decision between two reasonable page designs doesn't carry that cost. Running at 90% confidence, or even 85% for low-stakes decisions, meaningfully cuts required sample — provided you write the threshold down before launch and everyone understands the trade. What's not allowed is deciding the threshold after seeing the result.

Second: use evidence that isn't a split test

A/B testing is the gold standard for causation, but it's not the only admissible evidence, and pretending otherwise leaves low-traffic teams with nothing.

• Pre/post with a stable comparison. Ship the change, compare four weeks before and after, and control for seasonality with a metric the change couldn't have affected (another page, another product line). Weaker than a test, far better than nothing — and honest, if reported as what it is.
• Watch people. Ten recorded sessions or five moderated user tests will find the confusing form field, the mistrusted price, the dead-end mobile nav. At low traffic, qualitative research isn't the poor cousin of testing — it's a higher-yield instrument, because it finds problems you'd never think to test.
• Fix defects without ceremony. Broken layouts, contradictory prices, forms that error unhelpfully — these need a ticket, not a hypothesis. Reserve your scarce experimental traffic for genuine uncertainty.

Third: borrow scale where it exists

If you operate many similar low-traffic properties — a portfolio of location sites, for instance — you can sometimes test across them: same change, multiple sites, pooled analysis (with sites as the unit, which needs care but is doable). Individually untestable pages become a testable population.

The honest summary

Low traffic doesn't mean no experimentation. It means: fewer tests, bigger swings, upstream metrics, explicitly chosen thresholds, and a bigger role for qualitative evidence — all written down so nobody mistakes pragmatism for sloppiness. That programme is smaller than the one the brochure promised you. It also works, which the brochure programme, at your traffic, quietly doesn't.