← All articles

Growth · 11 min read · 2026-10-28

Incrementality Testing for SaaS: Geo Holdouts & Lift

An attribution model tells you which touchpoints a converter saw and how to split credit among them. It cannot tell you how many of those conversions would have happened anyway — and that is the only number a CFO deciding whether to keep funding a channel actually cares about. Incrementality testing answers the counterfactual directly: withhold the spend from a random control, run it for a treatment group, and measure the gap. The gap is the lift the spend caused, and it is almost always smaller than the credit an attribution model hands out. This is a playbook for running the two designs that work best for SaaS — geo holdouts and ghost-ad lift tests — and for feeding the result back into your attribution so the models stay honest.

The reason this matters is that attribution and incrementality routinely disagree, and teams that only run attribution never find out. The field has converged on triangulating multi-touch attribution, marketing-mix modelling, and incrementality rather than trusting a single model — incrementality is the leg that supplies ground truth. As idukki puts it, incrementality is the revenue that exists because of the activity and would vanish without it, measured by withholding the experience from a random group and comparing. Everything below is how to get that measurement on SaaS spend without fooling yourself.

Why attribution credit is not causal lift

Attribution has a structural blind spot: it only observes people who converted. Because it never sees the users who did not convert, it cannot know how many converters would have arrived with no spend at all, so it credits every channel on the winning path — including channels that merely intercepted people already on their way to buying. Branded search is the canonical offender: a model happily assigns conversions to a brand-name campaign that was reaching users who typed your name because they had already decided. That is why the branded-search lift question can only be settled by a holdout, not by reading a model's output.

Incrementality removes the blind spot by construction. With a randomised control that did not receive the spend, you observe the full base rate of conversion in the absence of the activity, and the difference between treatment and control is the causal effect. This is the same logic as any controlled experiment, and it is why incrementality results so often undercut the dashboard: the dashboard reports assigned credit, the experiment reports caused conversions. When the two numbers diverge, the experiment is the one to believe — a point worth keeping in mind whenever an attribution report looks too good to be decision-grade.

Geo holdouts: trade granularity for a clean control

A geo-holdout test partitions comparable regions into treatment and control, runs the campaign only in treatment regions, suppresses it in control regions, and compares outcomes. Its great virtue is contamination resistance: unlike a user-level holdout, treatment does not easily leak across a regional boundary, so the control stays clean even for channels you cannot target at the individual level. That makes geo tests the default design for measuring upper-funnel and brand spend whose effect is diffuse and hard to pin to individual users.

The price is statistical power. You need enough comparable regions and enough conversion volume in each to detect a lift above regional noise, and you must match treatment and control markets on size and baseline trend before you start or the comparison is meaningless. When well-matched markets are scarce, synthetic-control methods — Meta's GeoLift and Google's CausalImpact are the common open tools — build a counterfactual for each treated market from a weighted blend of untreated ones. Size the holdout and test duration the same way you would any powered experiment: decide the minimum detectable lift first, then compute how much volume and how many weeks that requires.

Ghost ads: the right control for branded and search

Audience holdouts break on branded search, because the audience self-selects: the people in it are defined by the very act of searching your brand, so any "control" you carve out of them is already different from the treated group. Ghost ads solve this by logging the users who were eligible to be served your ad and would have seen it, but were held out — an intent-to-treat control defined by eligibility rather than behaviour. You then compare conversion between the users who saw the ad and the identically-qualified users who did not, and the difference is incremental.

Ghost-ad and PSA designs are mainly a paid-search and large-platform capability, and the platforms gate them behind spend and volume minimums, so confirm eligibility before you plan around them. Where you can run one, it is the cleanest available answer to the branded-search credit problem and to any campaign where exposure correlates with intent. Treat vendor-quoted thresholds and holdout sizes as starting points to verify against current platform documentation, not as fixed rules — they change, and an underpowered ghost-ad test is as misleading as an underpowered geo test.

Reading the lift without fooling yourself

Report incremental lift with its uncertainty, never as a point estimate. A test that says "18% lift" means nothing without a confidence interval; if the interval spans zero, the honest conclusion is that you could not detect an effect, not that the effect is zero. The most common way teams deceive themselves is peeking — watching the result accumulate and calling the test the moment it crosses significance — which inflates false positives exactly as it does in product experiments. The discipline that controls the peeking problem applies here unchanged: fix the duration up front and read the result once.

Guard against the structural failures too. An underpowered test reports noise as signal, so if your conversion volume cannot support a conversion-based geo test, move the measurement to an upper-funnel proxy with more events — site visits, qualified leads, trial starts — rather than running a test that cannot see its own effect. Check that treatment and control did not drift apart for reasons unrelated to the campaign (a pricing change, a seasonal skew in one region), because a confound in the control quietly becomes fake lift. The goal is a lift number you would stake a budget decision on, not one that merely looks precise.

Feed the result back into attribution

An incrementality test is not a one-off verdict; it is a calibration input. Once you know the true incremental lift of a channel, you can compare it to the credit your attribution model assigned and compute a haircut — the factor by which the model over- or under-credits that channel. Applying that haircut to the model's ongoing output gives you day-to-day attribution you can trust between tests, without pretending the raw model was ever ground truth. This is the practical way to resolve the disagreement between MMM and MTA: let the experiment arbitrate and recalibrate both.

Build a cadence rather than a single test. Channels and creative fatigue, so a lift measured last quarter decays; re-test the channels that drive the most spend on a rolling schedule and treat the haircuts as living coefficients. Pair the incrementality programme with your multi-touch attribution model and with a discipline of discounting platform-reported ROAS down to ground truth, and you have the full triangle: models for daily credit, incrementality for causal truth, and a feedback loop that keeps the daily credit tethered to reality. That is what turns a budget decision from an argument about dashboards into a measurement.

FAQ

What is incrementality testing?

Incrementality testing measures the revenue or conversions that exist only because of a marketing activity and would disappear without it. You withhold the activity from a randomly chosen control group, run it for the exposed group, and compare outcomes; the gap is the incremental lift. Unlike an attribution model, which divides credit among touchpoints a converter happened to see, an incrementality test asks the counterfactual question directly — what would have happened if we had not spent — and so measures causal effect rather than assigned credit.

How is incrementality different from attribution?

Attribution distributes credit across the touchpoints on a converting user's path using a model (last-click, data-driven, multi-touch). It can only see people who converted, so it systematically credits channels that reach users who would have converted anyway. Incrementality compares a treated group against an untreated control, so it captures the users who did not convert and isolates the effect caused by the spend. Attribution answers 'which touchpoints did converters see?'; incrementality answers 'how many conversions did this spend actually create?' — a different and usually smaller number.

What is a geo-holdout test?

A geo-holdout test splits comparable geographic regions into treatment and control: you run the campaign in the treatment regions and suppress it in the control regions, then compare outcomes between the two sets. Geo tests trade user-level granularity for a clean, contamination-resistant control, because you cannot easily leak the treatment across a regional boundary. The cost is sample size — you need enough comparable regions and enough conversion volume per region for the lift to rise above noise, which is the main constraint for lower-volume B2B SaaS.

What are ghost ads and when do you use them?

Ghost ads (and the related PSA/placebo approach) log the users who would have been served your ad but were held out, creating an intent-to-treat control that saw nothing while still being identified by the same targeting. This removes the selection bias of comparing exposed users to a generic unexposed population. Ghost-ad designs suit paid-search and branded campaigns, where an audience holdout fails because users self-select by searching your brand name, so you need a control defined by eligibility rather than by behaviour.

Can early-stage SaaS run incrementality tests?

Often not with a conversion-based geo test, because low conversion volume per region leaves the test underpowered and it will report noise as signal. Lower-volume SaaS can instead run incrementality on an upper-funnel proxy with more events (site visits, qualified leads, trial starts), use a synthetic-control method that models a counterfactual from a weighted blend of untreated markets, or hold out a larger share of spend for longer to accumulate enough signal. If none of those produce a powered test, be honest that an incrementality claim is not yet available and lean on attribution with stated uncertainty.