Product · 9 min read · 2026-10-18
The Peeking Problem: How Early A/B Peeks Fake Wins
A growth team starts an experiment on a new onboarding step, opens the dashboard every morning, and on day nine sees the variant cross 95% significance with a +12% lift. They stop the test and ship. A month later activation is flat, and the "significant" win has vanished. Nothing broke in the pipeline. The team simply ran a different test than the one the p-value described: by looking every day and stopping on the first green, they turned a 5% false-positive rate into something closer to one-in-three.
This is an anti-pattern teardown of the peeking problem — the most common way a correctly instrumented, adequately powered A/B test still lies to you. It is a cousin of the time-decay issue in the novelty and primacy effect post and of the sizing math in the experiment sizing and test-duration guide, but it is a distinct failure: here the sample is fine and the effect is stable, and the error enters entirely through when and how often you look.
Peeking is repeated significance testing in disguise
A fixed-horizon test earns its 5% false-positive guarantee from one assumption that almost no one honours: you look at significance exactly once, at the sample size you committed to before the test began. The moment you check it on day three, again on day five, and again on day nine, you are running the hypothesis test multiple times on overlapping data, and the chance that noise crosses the threshold on at least one of those looks grows with every look. As one practitioner reference puts it, the test is simply not safe under optional stopping: deciding when to stop based on the result you can see changes the error rate.
The size of the inflation is larger than most teams guess. A common illustration holds that peeking daily during a month-long experiment can lift the false-positive rate from the standard 5% to upwards of 30%, and when you combine it with stopping early and testing many metrics at once, the rate can climb past 50%. Treat the exact figures as illustrative rather than precise, but the direction is not in doubt: every extra look spends error you did not budget for, and a dashboard that updates a "significant" badge in real time is inviting you to spend all of it.
Early stops also exaggerate the lift
Peeking does not only manufacture wins out of nothing; it inflates the size of the wins that are real. When you stop the instant significance appears, you are selecting the moment the measured lift happened to spike on a lucky upward swing, because that is precisely when the threshold is easiest to cross. The result is that your reported effect is captured at a local peak rather than at its settled value, so even a genuine +4% change can be booked as +12% because you froze the number on its best day. This is the winner's-curse mechanism behind the regression-to-the-mean trap: an extreme reading is extreme partly because it is lucky, and luck does not ship.
That is why peeking damage survives even when the change works. The team that stopped on day nine may well have a real winner, but they have no idea whether the durable effect is +12%, +4%, or +1%, which means their forecast of the change's impact — the number that justified the engineering time — is built on the single most optimistic estimate the test ever produced. Reading an experiment the moment it looks best is a close relative of confusing correlation with causation: the figure is arithmetically real and still points you at the wrong decision.
Fix one: pre-declare a fixed horizon and do not look
The cheapest fix requires no new statistics, only discipline. Compute the required sample size up front from your baseline rate, your minimum detectable effect, and your power target — the same math in the experiment sizing guide — then commit to reading significance exactly once, when you hit that sample. Between launch and the horizon you are allowed to monitor health (is the split balanced, are events flowing, has anything broken) but not the verdict. The decision look is a single event at the end, and the 5% guarantee holds because you only spent it once.
The practical obstacle is organisational, not mathematical: someone will open the dashboard, see green, and ask to ship. The defence is to hide the interim verdict by default — show sample progress and a guardrail health check, but not a live p-value or a "significant" pill — so the only time the win/lose readout appears is at the pre-declared horizon. Pair the stopping rule with a sample-ratio-mismatch check at that same final look, so the one decision point validates the split before it trusts the lift.
Fix two: use a design built for continuous looks
If you genuinely need to watch a test and stop early — because a loser is costing money, or because traffic is scarce and you cannot afford to over-run — then use a method designed for exactly that instead of abusing a fixed-horizon test. Group-sequential designs with an alpha-spending function (O'Brien-Fleming or Pocock) carve your 5% error budget into pre-planned interim analyses, so each look spends only a slice and the overall guarantee survives. Always-valid approaches such as mSPRT sequential testing go further and remain valid at every look, letting you monitor continuously and stop the moment the evidence is conclusive.
The trade-off is honest and worth stating: anytime-valid inference is not free. A sequential test needs a larger sample to declare the same effect significant than a fixed-horizon test that looks once, because you are buying the right to peek with error budget spent across many looks. For a high-traffic flow that is a bargain; for a low-traffic SaaS experiment the extra sample may be more than you can gather in a reasonable window, in which case the fixed-horizon-plus-hidden-verdict approach is the more realistic discipline. The one combination that is never valid is a fixed-horizon test monitored like a sequential one.
Why SaaS experiments are especially exposed
Peeking is a worse hazard for B2B SaaS than for a high-volume consumer app, for a structural reason: your tests run long. A flow that a consumer product powers through in two days can take a SaaS team three weeks to adequately power, and every one of those calendar days is a day someone checks the dashboard. The long runway multiplies the number of looks, and the small per-day sample makes each interim estimate noisier and more likely to throw a misleading spike that tempts an early call.
The compounding factor is metric count. A SaaS experiment rarely tracks one number; it tracks activation, conversion, a revenue proxy, and two or three guardrails, and watching all of them for the first to turn green is multiple-comparisons peeking stacked on top of optional-stopping peeking. The disciplined setup pre-declares a single primary metric, treats the rest as guardrail counter-metrics rather than co-winners, and decides the test on that one metric at the horizon — so the breadth of your instrumentation stops working against the validity of your conclusion.
A peeking guardrail you can run this sprint
Before your next experiment starts, write down three things and hold the team to them. One: the pre-declared sample size and the single primary metric the decision rests on. Two: whether this is a fixed-horizon test (verdict read once, at the horizon) or a sequential test (valid at every look) — and never a blend of the two. Three: what is visible on the dashboard before the horizon, which should be split health and sample progress, not a live significance verdict. If your tool cannot hide the interim verdict, the human fix is a rule that nobody calls a test outside the weekly experiment review.
None of this needs a statistics team — it needs the stopping rule decided before launch rather than improvised the morning a bar turns green. The teams that get burned by peeking are not short on data; they are the ones whose dashboard rewards the first green and whose culture treats significance as a finish line to sprint toward. Make the stopping rule a mandatory field on every experiment brief, the same way the novelty-decay check is, and the lucky-day win stops making it into your roadmap.
Frequently asked questions
The questions below cover what the peeking problem is, why early stopping exaggerates the effect size, how to avoid it with fixed-horizon or sequential designs, why low-traffic SaaS tests are especially exposed, and whether a sequential test really lets you stop early safely.
FAQ
What is the peeking problem in A/B testing?
The peeking problem is the inflation of your false-positive rate that happens when you repeatedly check an experiment's significance and stop as soon as it crosses the line. A standard fixed-horizon test controls the Type I error at 5% only if you look once, at the pre-declared sample size. Every extra look is another chance for random noise to cross the threshold, and because you stop the moment it does, you are selecting for the lucky swing. Peeking daily over a month-long test can push the probability of a false positive from 5% to around 30%, and combining it with testing many metrics at once can push it past 50%. The statistics are not broken; you have just run a different, much less stringent test than the one the p-value assumes.
Why does stopping an A/B test early exaggerate the effect size?
Because an early crossing of the significance line is biased toward happening on an upward random swing. When the true effect is small or zero, the measured lift wanders around its real value; the moments it spikes high are exactly the moments a peeking test is most likely to cross significance and get stopped. You therefore capture the metric at a local high rather than at its settled value, so the reported lift is systematically too large even in the cases where a real effect exists. This is why teams that stop early often see the shipped change underperform its test number: the test measured a lucky peak, not the durable effect.
How do you avoid the peeking problem?
There are two clean options. The first is a fixed-horizon test: pre-declare the sample size using a power calculation, do not read significance before you reach it, and make the end-of-test look the only decision point. The second is a sequential design built for continuous monitoring — group-sequential methods with an alpha-spending function (O'Brien-Fleming, Pocock) or always-valid inference such as mSPRT — which budget your error across many looks so you can watch the test daily and stop early without inflating the false-positive rate. What you cannot do is run a fixed-horizon test and monitor it like a sequential one; that combination is the peeking problem by definition.
Is peeking a problem for low-traffic SaaS experiments?
Yes, and often worse than for high-traffic consumer products. Low-traffic SaaS tests already run for weeks to reach power, which means more calendar days during which someone checks the dashboard and is tempted to call it the moment a bar turns green. The long runway multiplies the number of looks, and the small sample makes each interim estimate noisier and more prone to a misleading spike. If a dashboard shows a live p-value or a 'significant' badge that updates continuously, it is structurally encouraging the exact behaviour that breaks the test, so the safest setups hide the interim verdict until the horizon is reached or switch to a sequential method that is valid at every look.
Does a sequential test let me stop early safely?
Yes — that is the entire point of a sequential design, and it is the only legitimate way to stop early. Methods like mSPRT and group-sequential alpha-spending are constructed so that the error guarantee holds no matter when you stop, by spending a fraction of your error budget at each look rather than the full 5% at the end. The trade-off is that you pay for the flexibility: an anytime-valid test needs a larger sample to declare the same effect significant than a fixed-horizon test that looks only once. You are buying the right to peek and stop early with a modest increase in required traffic, which for many teams is worth it.