A/A tests: the cheapest credibility check your programme isn't running

An A/A test is exactly what it sounds like: you run your testing tool with two identical variants. Same page, same everything, traffic split 50/50. The correct result is a shrug — no significant difference, because there is no difference.

It sounds like a waste of a test slot. It's actually the closest thing experimentation has to calibrating the instrument before you fly the plane.

What an A/A test catches

Broken randomisation and bucketing. Users should land 50/50 (within noise) and stay in their bucket across sessions. We've seen tools assign a returning visitor to a different arm on each visit, redirect-based splits that leak one variant to search crawlers, and integrations that bucketed by session instead of user. Every one of those quietly corrupts every real test you run.

Sample ratio mismatch, before it matters. If your 50/50 split comes back 54/46 on real volume, something is interfering — bot filtering hitting one arm, a QA redirect, a consent banner behaving differently. Better to find that on a test where the answer doesn't matter.

Tracking asymmetries. Identical pages should record identical conversion rates. If variant "B" of the same page converts 8% higher, your analytics event is firing differently between arms — a flicker, a timing issue, a duplicated tag. This is depressingly common with client-side tools.

Your own statistical hygiene. Here's the uncomfortable one. Run the A/A, check it daily, and watch it go "significant" at some point mid-flight — as it will, because that's what peeking does. Nothing teaches a stakeholder the peeking lesson faster than a test between two identical pages "winning". We've converted more executives to statistical discipline with one A/A test than with any amount of explanation.

What it costs, honestly

The objection is real: an A/A test occupies traffic that could power a real experiment, and on a low-traffic site that's not nothing. Three answers:

1. You don't need significance-level sample sizes to catch gross problems. A fortnight usually exposes bucketing errors, SRM and tracking asymmetry — the failures big enough to matter.
2. You can often run an A/A alongside the programme as an A/A/B: two identical controls plus one variant. If the two A arms disagree with each other, you've learned your B result is untrustworthy — on the same traffic.
3. Weigh it against the alternative: a corrupted platform silently invalidating a year of decisions. As insurance goes, one test slot is cheap.

When to run one

• When you adopt a new tool or migrate platforms. Non-negotiable, before the first real test.
• After major site changes — a replatform, a new consent management setup, a framework migration. Anything that touches how pages load can touch how experiments render.
• When results have started looking too good. A run of implausible winners is more often a broken instrument than a hot streak.
• Annually, as a matter of routine, the way you'd service a boiler.

An experimentation programme's entire value rests on one assumption: that the numbers coming out of the tool describe reality. An A/A test is the only procedure that checks the assumption itself. Run one before your next big test, and you'll either gain confidence or find a problem — and both of those are wins.