Product · 10 min read · 2026-10-12
Sizing a SaaS Experiment: MDE, Power, and Duration
Most failed SaaS experiments were doomed before they launched, because nobody asked the one question that decides whether a test can work: how much traffic will this take, and how long will it run? The answer comes from three inputs you set in advance — the minimum detectable effect, the statistical power, and the significance level — and your real traffic. Get them right and the test gives a clean verdict. Skip the exercise and you ship an underpowered test that returns "no significant difference" whether or not a real effect exists.
This is a sizing playbook: how to choose each input, how they trade off, and how to convert the required sample into a duration from your own signups per day. It is the planning step that sits upstream of the test, and it is distinct from running an experiment with tiny volume at all — if you are traffic-starved, start with low-traffic experimentation tactics first, then come back here to size what is feasible.
MDE: the smallest win worth shipping
The minimum detectable effect is the smallest true difference between variants you want the test to catch reliably. It is as much a product decision as a statistical one: set it to the smallest improvement that would actually change what you do. If a 1% lift in trial-to-paid would not justify shipping and maintaining the variant, do not design a test to detect 1% — the cost is enormous. If a 5% lift would clearly be worth it, that is your MDE. The number should fall out of a conversation about what's worth your engineering time, not out of a tool's default box.
MDE is quoted as either an absolute or a relative change, and conflating the two is a common sizing error. A baseline conversion of 4% lifted to 5% is a one-point absolute change but a 25% relative change, and a calculator will give very different sample sizes depending on which you enter. Pick one convention, state it explicitly, and make sure your baseline rate is measured on the same qualifying population you will test — which is exactly the denominator-choice problem applied to experiment design. A baseline computed over the wrong base throws the entire sizing off from the first step.
Power and significance: the two error rates
Statistical power is the probability the test detects a real effect of at least your MDE when one genuinely exists; the convention is 80%, which accepts a 20% chance of missing a true effect. The significance level, usually 5%, is the complementary error — the chance of declaring a difference that is not real. These two numbers are the knobs on your two ways of being wrong, and both cost sample size: higher power levels demand more data to detect the same effect, and stricter significance thresholds push the required sample up as well (SplitMetrics on MDE).
For most SaaS product tests, 80% power and 5% significance is the right starting pair, and you only move off it for a reason. Raise power to 90% when a missed win is especially costly — a change you will not get another chance to test — and accept the longer run. Tighten significance below 5% when a false positive would trigger an expensive, hard-to-reverse rollout. The point is that these are deliberate trade-offs against test length, not universal constants, and a change you make to one has to be paid for in sample size somewhere else.
From inputs to sample size
With baseline rate, MDE, power, and significance fixed, a standard sample size calculator returns the number of observations you need per variant. The relationship worth internalizing is that sample size scales roughly with the inverse square of the MDE: for a fixed significance and power, a smaller MDE requires a larger sample, and a 5% relative MDE can need about four times the sample of a 10% MDE (Personizely). That non-linearity is why "let's just detect a tiny change" is rarely free — halving the detectable effect roughly quadruples the cost.
Lower-baseline metrics also cost more sample. A test on a 2% conversion surface needs far more traffic to detect the same relative lift than a test on a 40% surface, because rare events carry more relative variance. This is the practical reason you cannot test everything: deep-funnel metrics with low base rates demand sample volumes most SaaS products simply do not have at the leaf of the funnel, which is why many teams test higher-base upstream steps and infer downstream — accepting the assumptions that requires. Choosing the metric you can actually power is itself part of sizing.
Turning sample size into weeks
Sample size becomes a schedule when you divide it by daily qualifying traffic per variant. The rule is simple: if you need 15,000 users per variant and receive 500 qualifying users per day to each arm, the test runs 30 days; with an expected 3% baseline, a 0.5% absolute MDE, 5% significance, and 80% power requiring roughly 17,000 per group at about 5,000 users a day, it runs about a week (Evan Miller's calculator). The number that trips teams up is "qualifying" — count only the users who actually reach the surface under test, not total signups or total visitors, or your duration estimate will be wildly optimistic.
Round the result up to whole weeks. SaaS traffic and conversion vary sharply by day of week, and a test that runs 17 days covers two and a half weekends' worth of one composition and not the other, letting day-of-week mix leak into the comparison — the same distortion that corrupts naive period-over-period metric comparisons. Running in whole weeks balances the composition on both arms. If the honest duration comes back at three months, that is the sizing telling you the test is not viable as designed — raise the MDE, pick a higher-base metric, or do not run it.
Why you commit before you start
The sample size and MDE are not just a planning convenience — fixing them before the test runs is what makes the result statistically valid. The failure mode is peeking: watching a running test and stopping the moment it crosses significance. Each look is another chance to cross the threshold by luck, so repeated peeking at a fixed-horizon test inflates the real false-positive rate far above the nominal 5%, and a "significant" result obtained that way is often noise. Committing to a sample size up front, and reading the verdict only once it is reached, is what keeps the stated significance honest.
If you genuinely need to monitor a test in flight — to kill a harmful variant early — use a sequential testing method built for repeated looks rather than eyeballing a standard test. And pair the primary metric with guardrail metrics so a win on the target cannot quietly come at the cost of something you care about more. Validity also depends on the split itself being clean, which is why a sample ratio mismatch check belongs in every test — a skewed allocation invalidates the sizing you did no matter how carefully you did it.
A sizing checklist for this week
Before the next test leaves the backlog, run it through five steps: measure the baseline rate on the exact qualifying population, set the MDE to the smallest lift worth shipping, choose power and significance (default 80% and 5%), compute the required per-variant sample, and divide by daily qualifying traffic to get a duration. Round up to whole weeks. If the answer is feasible, you have a test worth running; if it is three months, you have saved yourself from running an experiment that would have returned a meaningless null.
Do this sizing once as a team and it changes which experiments you even propose, because the cost of each becomes visible before engineering time is spent. The teams that ship reliable wins are not the ones with the most tests running — they are the ones that only run tests they can actually power, and that read the result exactly once, at the sample size they committed to before they began.
Frequently asked questions
The questions below define the minimum detectable effect and statistical power, explain how MDE drives the sample size you need, show how to convert that sample into a test duration from your real traffic, and explain why fixing the MDE and sample size before the test runs is what keeps the result valid.
FAQ
What is the minimum detectable effect (MDE)?
The minimum detectable effect is the smallest true difference between variants that your test is designed to catch reliably. You choose it before the test starts, and it drives everything else: a smaller MDE means you want to detect a subtler change, which requires more data. It is a business decision as much as a statistical one, because it should be set to the smallest improvement that would actually be worth shipping. Setting the MDE too small commits you to an impractically long test; setting it too large means a real but modest win can go undetected and get discarded as noise.
How does MDE affect the sample size you need?
For a fixed significance level and power, a smaller MDE requires a larger sample, because detecting a subtler difference takes more evidence to separate from random variation. The relationship is steep and roughly inverse-square: a 5% relative MDE can need about four times the sample of a 10% MDE. This is why halving the effect you want to catch does not double the test length — it roughly quadruples it. The practical consequence is that the MDE is the single most powerful lever on how long a test runs, so it deserves deliberate choice rather than being left at a tool's default.
What is statistical power and what value should I use?
Statistical power is the probability that the test detects a real effect of at least the MDE, when one truly exists. The convention is 80%, meaning you accept a 20% chance of missing a real effect. Higher power — 90% — reduces that miss rate but demands more data and a longer test. Power trades off against the false-positive rate (the significance level, usually 5%) and against sample size: tightening either the power or the significance threshold increases the sample required. For most SaaS product tests, 80% power with a 5% significance level is the standard starting point.
How do I turn the required sample size into a test duration?
Divide the required sample size per variant by your daily traffic per variant. If a variant needs 15,000 users and you send 500 qualifying users per day to each arm, the test runs about 30 days. Always compute duration from the qualifying population — the users who actually reach the surface you are testing — not total signups or total visitors, or you will badly underestimate how long it takes. Then round up to whole weeks so the test covers the same mix of weekdays and weekends, which avoids day-of-week composition skewing the result.
Why must I fix the MDE and sample size before the test runs?
Fixing them up front is what makes the result valid. If you watch the test and stop as soon as it looks significant, you inflate the false-positive rate dramatically — this is the peeking problem, and it turns a nominal 5% error rate into something far higher. Committing to a sample size and an MDE before you start, and only reading the result once that sample is reached, is what keeps the stated significance honest. If you genuinely need to monitor along the way, use a sequential testing method designed for repeated looks rather than naively peeking at a fixed-horizon test.