What is the novelty effect?

Why does the novelty effect distort tests?

Returning visitors have an expectation of your site. A changed layout, a moved button, or a new color violates that expectation and draws attention. Attention produces interaction, and interaction looks like improvement.

The distortion is largest exactly where teams are most eager: the first few days of a test, on a site with a high proportion of returning visitors. A variant showing a 30% lift on day two and a 3% lift on day fourteen was never a 30% variant. It was a 3% variant with an audience noticing a change.

The damage compounds because early results are what tempt teams to stop early, and stopping early is what locks in the inflated number.

How do you detect it?

Three checks, cheapest first.

Plot the effect over time, rather than the cumulative total alone. A genuine improvement produces a roughly stable difference across the test window. Novelty produces a difference that starts high and decays. The shape of the curve is the tell, and the cumulative number hides it.

Segment new versus returning visitors. Novelty acts on people with a prior expectation. If the lift appears among returning visitors and vanishes among new ones, it is novelty. If it holds among new visitors, it is a real effect. This is the check that settles the question.

Run long enough for the audience to cycle. A test covering less than one full visit cycle for your returning population cannot separate the two.

What is the opposite problem?

Change aversion, and it is the mirror image.

Returning visitors accustomed to a workflow perform worse when it changes, regardless of whether the new version is better. The variant loses in week one, recovers in week two, and would win in week four. Teams that stop early kill genuinely better designs.

Both effects come from the same source, which is that existing users have habits. Novelty inflates changes that attract attention, change aversion penalizes changes that disrupt a learned path. Neither is present among first-time visitors, which is why the new-versus-returning segment resolves both.

How do you design around both?

1. Set the duration before launch, at one full business cycle minimum, and longer on sites where people return every few weeks rather than every few days. The window has to be long enough for your returning population to see the change more than once.
2. Declare the new-versus-returning split in advance, so the check above counts as evidence rather than as segment mining after a disappointing read.
3. For large interface changes, keep a holdout. A small group left on the old version after shipping tells you a month later whether the lift survived contact with habit. See holdout group.
4. Weight new-visitor results more heavily when the change targets acquisition, since new visitors are the population the change is meant to serve and they carry neither bias.

Related terms

Statistical significance · Holdout group · Sequential testing · Guardrail metrics · Split testing

Deeper reading: What is structured A/B testing. Service: Conversion Rate Optimization.

FAQ

How long does the novelty effect last?

It depends on visit frequency, since it decays as returning visitors re-encounter the change. On sites where people return weekly it can persist for several weeks. On sites where most traffic is new it barely exists.

Does the novelty effect apply to new visitors?

No. A first-time visitor has no prior expectation to violate, so there is nothing novel to react to. This is why segmenting by new and returning is the definitive test.

Can the novelty effect make a bad variant win?

Yes, and it is the main practical risk. A variant that is worse on merit can outperform for the first days because it draws attention, then underperform once attention normalizes. If the test stopped during the first phase, the worse variant ships.

Is change aversion the same as the novelty effect?

They are opposites with a shared cause: existing users have expectations, and any change violates them. Novelty inflates early results and change aversion depresses them, so an early read can either ship a weak variant or kill a strong one.