Product · 9 min read · 2026-10-17
How the Novelty Effect Fakes Your SaaS A/B Test Wins
A team ships a redesigned dashboard to half their users, watches engagement jump 18% in the first week, calls the test a win, and rolls it out to everyone. Six weeks later engagement is flat against the old baseline, sometimes below it, and nobody can explain why the "proven" change did not move the metric. Nothing went wrong with the rollout. The 18% was never the treatment effect — it was the novelty effect, a curiosity spike that was always going to decay, and the test was stopped at the exact moment the illusion looked strongest.
This is an anti-pattern teardown of the single most common reason shipped experiments fail to replicate in the metric later: the result was read as a flat number when it was really a curve. The blog's experiment sizing and test-duration guide tells you how long to run for statistical power; this post is about the validity threat that survives even a perfectly powered test — time decay in the effect itself.
Novelty and primacy are the same bug with opposite signs
The novelty effect and its mirror image, the primacy effect (also called change aversion), both come from the same place: existing users reacting to the fact that something changed, not to whether the change is good. A LogRocket breakdown describes novelty as engagement that rises out of curiosity and then drops back toward the real baseline as newness wears off. Primacy runs the other way: a new layout or workflow initially hurts task completion because people have to relearn habits, and the variant looks worse than it is until they adapt.
The consequence for your test is that the first-week number is biased in an unknown direction. A visible, exciting change — a new feature surface, a redesigned nav — tends to get a novelty bump and look like a false winner. A change that disrupts a practiced flow — a moved button, a reordered settings page — tends to get a primacy penalty and look like a false loser, which is how teams kill changes that would have won in a month. Either way, the pooled lift you read at the end of the test is an average of a decaying or recovering curve, and the average hides the shape that actually decides whether you should ship.
Why the pooled lift lies to you
The default output of almost every experimentation tool is one number: the relative lift, pooled across the whole test window. That number implicitly assumes the treatment effect is constant over time, which is precisely the assumption novelty and primacy violate. If the true effect is +18% in week one and 0% from week three onward, the pooled estimate lands somewhere in the middle — say +6% — and reports a win that does not exist at steady state. The statistics are not wrong; significance testing confirms the +6% is real for the window measured. The window is just not the window you care about.
This is why novelty is a validity threat and not a sample-size problem. Running the test on more users makes the +6% more precise without making it more correct, the same way a tighter confidence interval around a mismeasured quantity does not fix the mismeasurement. It belongs to the same family of silent interpretation errors as confusing correlation with causation: the number is defensible on its own terms and still points you at the wrong decision, because the thing being averaged is not stable over the period you will actually live with the change.
Detection one: trend the effect, do not pool it
The first detection move costs nothing and is the one most teams skip: compute the treatment effect per day or per week across the test, and plot it, instead of reading a single pooled figure. A durable effect is roughly flat — week one looks like week three. A novelty effect is front-loaded and declines each week; a primacy effect starts negative or small and climbs. The shape, not the endpoint, is the signal. Practitioner guides on novelty detection converge on exactly this: segment the metric by experiment week and look at the week-over-week trend before trusting the aggregate.
You do not need an exotic test for this. Eyeballing the weekly series catches the obvious cases, and a simple monotonic-trend check (a rank correlation of effect against week) formalizes it when you want a decision rule. The point is to make "is this effect stable?" a question your readout answers by default, the same discipline that day-of-week seasonality forces on any short-window comparison. If the effect is still visibly sliding at the end of the test, the test is not done, whatever the p-value says.
Detection two: split new users from returning users
The second move exploits the fact that novelty and primacy only affect people who have seen the old version. New users have no prior to be surprised by or to resist, so they are a built-in control for the effect. Segment the experiment results by tenure: if returning users show a big early lift that fades while new users show a smaller, stable effect, the aggregate was inflated by returning-user curiosity. The reverse pattern — new users fine, returning users depressed early then recovering — is the fingerprint of change aversion, not a bad feature.
This split also tells you what to do next, which the pooled number never does. If new users like the change and returning users are merely change-averse, the right call is usually to ship and let the base adapt, possibly with in-product guidance to shorten the relearning period. If new users are indifferent and the entire lift lived in returning-user novelty, the change has no durable value and shipping it just spends engineering time for a metric that will settle back. Reading the two cohorts separately is the same anonymous-versus-known discipline behind pre-signup activation instrumentation: the aggregate blends populations that are reacting to completely different things.
Design the test so decay cannot fool you
Detection is the floor; the better teams design the novelty window out of the decision up front. The cheapest design fix is duration: run visible changes at least two to three weeks past first significance so a one-to-two week novelty cycle has time to decay inside the measured window, and set the stopping rule on "the weekly effect has flattened," not on "the p-value crossed 0.05." How long that takes depends on usage cadence — a daily-use tool burns through novelty in days, while a tool people open weekly can carry a curiosity bump for a month before it fades.
For high-stakes changes, keep a small long-run holdout: a slice of users who stay on the control for weeks or months after you ship to everyone else. The holdout gives you a clean read on the durable effect long after the novelty has washed out, and it is the only method that measures the steady state directly rather than inferring it from a trend. A long-run holdout is a guardrail in the same sense as an explicit counter-metric: a small, permanent cost you pay so a confident-looking launch cannot quietly be a loss. Alternatively, if the change only needs to work for new users — an onboarding tweak, say — run it on new users alone and sidestep both effects entirely.
A novelty check you can run this sprint
Before you call your next A/B test a win, run three checks. One: pull the treatment effect by week and look at the shape — flat is real, sliding is novelty, climbing is primacy. Two: split the result by new versus returning users and ask whether the lift survives in the group that cannot feel novelty. Three: confirm the test ran long enough that the final week sits inside a flattened trend, not on the steep part of the curve. If any check fails, the honest move is to keep the test running, not to ship on the early number.
None of this requires new tooling — it requires reading the experiment as a time series instead of a single figure, which is a reporting choice you can make today. The teams that get burned by novelty are not the ones without the data; they are the ones whose readout pools it away. Treat the per-week trend and the tenure split as mandatory columns on every experiment scorecard, the same way a sample-ratio-mismatch check is mandatory before you trust the split at all, and the fake week-one win stops making it into your roadmap.
Frequently asked questions
The questions below cover what the novelty effect is, how it differs from the primacy effect, how to detect decay in an experiment, how long to run a test to avoid novelty bias, and why new-user experiments are immune.
FAQ
What is the novelty effect in A/B testing?
The novelty effect is a temporary engagement spike that existing users show toward anything new, driven by curiosity rather than durable value. A redesigned button, a new tab, a changed flow all get clicked more in the first days simply because they are unfamiliar and users explore them. That spike decays back toward baseline as the newness wears off, usually within one to two weeks. The danger is reading the spike as the treatment effect: you ship the variant on the strength of a week-one lift, the curiosity fades, and the metric settles at or below where it started. Novelty inflates early results for changes that are visible to returning users, which is most UI and navigation changes.
What is the difference between the novelty effect and the primacy effect?
They are opposite distortions with the same root cause: existing users reacting to change rather than to value. Novelty makes a variant look better than it is early on, because curiosity drives extra engagement that later fades. The primacy effect (also called change aversion) makes a variant look worse early on, because users must relearn a familiar workflow and resist the disruption before they adapt. Novelty decays a positive result downward over time; primacy recovers a negative result upward over time. Both mean the week-one number is not the durable number, and both are invisible if you only read the aggregate lift at the end of the test.
How do you detect a novelty effect in an experiment?
Trend the treatment effect over time instead of reading one pooled number, and split by user tenure. First, compute the per-day or per-week lift across the test window: a genuine effect is roughly flat, while a novelty effect is large in week one and shrinks each week after. Second, segment new users from returning users. New users have no prior version to be surprised by, so they are immune to both novelty and primacy; if the lift holds for new users but collapses for returning users, the aggregate was novelty-driven. Microsoft's experimentation team has reported that the first-week estimate falls outside the longer-run estimate in a meaningful share of experiments, far more often than random noise would predict.
How long should you run an A/B test to avoid novelty bias?
Long enough for the novelty window to pass and the per-week effect to stabilize, which for visible SaaS changes usually means running at least two to three weeks past the point you hit statistical significance, not stopping the moment the p-value crosses the line. The exact duration depends on your usage cadence: a tool users open daily burns through novelty faster than one they open weekly, where a single novelty cycle can span a month. The discipline is to decide the window by when the weekly effect flattens, not by when significance first appears, and to keep a long-run holdout for high-stakes changes so you can measure the durable effect directly.
Does the novelty effect apply to new-user experiments?
No, and that is exactly why new users are the clean comparison group. Novelty and primacy are reactions to change, and a first-time user has no previous version to find novel or to resist. Running an experiment only on new users removes both effects by construction, which is why onboarding and activation tests are often more trustworthy than tests on established users. The trade-off is that new-user results do not tell you how your existing base will react, so a change that must roll out to everyone still needs a tenure split or a holdout to separate the curiosity bump from the real effect on the people who already use the product.