Sample Ratio Mismatch: The A/B Test Validity Check

You design an A/B test for a clean 50/50 split, let it run to significance, and the variant wins by four points. Ship it? Not yet — first check the counts. If control drew 51,200 users and the variant drew 48,800, you may not have a winning variant at all; you may have a broken experiment. That gap is sample ratio mismatch, and it is the single fastest way to catch an A/B test that is quietly lying to you. Most SaaS teams never run the check, which is why they act on results that a two-second test would have thrown out.

This is a mechanics post, in the same vein as the traps that govern experimentation on low-traffic SaaS. There the enemy is too little data; here the enemy is data you should not trust even when you have plenty of it. SRM does not care how large your sample is — a broken split invalidates a result at any volume, and the only defense is to look for it every single time.

What sample ratio mismatch is

Sample ratio mismatch means the observed allocation of users between your test arms differs from the intended split by a statistically significant margin. Random assignment never lands exactly on 50/50 — a little wobble is expected and harmless. SRM is when the wobble is too large to be chance: the counts have drifted further apart than fair coin-flipping would produce, which tells you a non-random force is at work. As Analytics Toolkit defines it, SRM is when the split "statistically significantly differs from the specified allocation proportions" (Analytics Toolkit).

Why does that invalidate the test? Because A/B testing rests on one assumption: the two groups are identical except for the change you made, so any difference in outcome is caused by that change. SRM is direct evidence the assumption has failed. If some mechanism is pushing users unevenly into the arms, that same mechanism is almost certainly correlated with the metric you are measuring — and now you cannot tell whether the variant "won" because it was better or because the broken split loaded it with a different kind of user. The result is confounded, and a confounded result is not a result.

The chi-square check, in one step

Detecting SRM is a single chi-square goodness-of-fit test comparing the observed user counts against the counts you expected from your intended split. Feed it two numbers per arm — observed and expected — and it returns a p-value: the probability of a split at least this uneven if assignment were truly fair. A high p-value means the imbalance is ordinary noise; a very low one means the split is broken. For a designed 50/50 test with 100,000 total users, expected is 50,000 per arm, and you compare that to what you actually got.

The standard alarm threshold is p < 0.01. As Analytics Toolkit puts it, exposures are considered unbalanced "if the SRM p-value is below 0.01," and an observed p-value under 0.001 is, in their words, "solid ground for suspecting the trustworthiness of the test outcome" (Analytics Toolkit). Pick one cutoff, apply it automatically to every experiment, and treat a failing check as a hard stop. The whole test costs seconds and needs nothing but the raw counts you already have — there is no excuse for skipping it.

Why a 51/49 split can be fatal

The counterintuitive part is that a split that looks trivially close — 51/49, even 50.5/49.5 — can be a screaming failure at scale. Chi-square is sensitive to sample size: the more users, the smaller the percentage gap needed to fall below your p-value threshold. At a few hundred users, 51/49 is noise; at a million, it is a five-alarm SRM. This is why "it's basically 50/50, who cares" is exactly the wrong instinct. The eye cannot judge SRM, because whether a given imbalance is significant depends entirely on the volume behind it — which is what the chi-square test exists to compute.

And a small percentage imbalance is not a small problem. If a 1% skew in counts is being produced by, say, a slow-loading variant that sheds impatient users, then the users missing from that arm are not random — they are the impatient ones, precisely the segment whose conversion behavior differs. A tiny gap in the ratio can correspond to a large, systematic difference in who is in each group. The magnitude of the count mismatch understates the damage, which is why you react to the significance of the mismatch, not its size.

What breaks the split

The usual culprits share a signature: something treats the two arms differently before your metric is measured. A variant behind a redirect or an extra network call loads slower and loses users who bounce during the delay. Bot and invalid traffic hits one arm more than the other. The bucketing logic has a bug, or a downstream filter drops users from one group but not its twin. Tracking that fires inconsistently undercounts one variant. Each of these injects a non-random difference upstream of the outcome.

In SaaS the latency version is the most common and the most insidious, because it looks like the variant is simply performing differently when in fact it is selectively losing users. Any variant that adds a round-trip, gates behind a feature flag evaluated late, or renders more slowly will shed a biased slice of traffic — and that bias masquerades as a conversion effect. The lesson is that SRM is usually not a statistics problem but an engineering or instrumentation problem surfacing through the counts, which is why the fix lives in the pipeline, not the analysis.

What to do when a test fails the check

Discard the result — do not try to rescue it. A test with SRM has non-comparable groups, so the conversion difference is confounded with whatever broke the split, and no amount of downstream analysis separates the two. As the practitioners at Analytics Toolkit put it, SRM "means the results are untrustworthy and should not be acted upon" and the experiment will "likely need to be discarded" until the cause is found (Analytics Toolkit). Reweighting or trimming the arms back to 50/50 is a trap: it hides the symptom while the mechanism that caused it keeps distorting your metric.

Instead, treat SRM as a bug report. Trace where users were lost or misassigned — compare arrival counts at each stage of the funnel, check the randomization, look for the slow path or the filter that only touches one arm. Fix that, then relaunch the experiment clean. Bake the chi-square check into your experiment readout so no result is ever reported without it, the same way a data-quality audit gates every other number you trust. An A/B program without an SRM check is shipping decisions on splits it never verified — and a broken split is worse than no test at all, because it wears the costume of evidence.