What is Bayesian A/B testing?

How does it differ from frequentist testing?

The two answer different questions using the same data.

The frequentist framework controls error rates across many hypothetical repetitions of the experiment. The Bayesian framework updates a belief in light of evidence. Neither is more correct. They are different tools, and the Bayesian output happens to match the question business stakeholders are actually asking.

What does a Bayesian result actually say?

"92% probability B beats A" means: given the data collected and the prior assumed, there is a 92% chance B's true conversion rate exceeds A's.

Two things that phrasing hides.

Every result carries a prior. A prior is a starting assumption about plausible effect sizes. Most platforms apply a weakly informative default, which is a sound choice and an invisible one. With small samples that assumption drives the answer substantially, and few teams ever inspect the one their tool uses.

Probability to beat says nothing about size. A variant with 92% probability to beat control and a likely lift of 0.2% is not worth the deploy. The credible interval on the effect size is where the useful number lives, and the readability that makes a probability easy to act on is the same property that makes a trivial effect easy to ship by mistake.

Does Bayesian testing let you stop early?

Watching is safe. Stopping on what you watched is where the claim gets oversold.

A posterior probability is a valid statement about the evidence at the moment it is computed, so a Bayesian readout does not accumulate the error that repeated looking inflicts on a fixed-horizon frequentist test. Sequential testing solves the same problem from the frequentist side.

Ending the test the moment the number looks good is a separate matter. A threshold gets crossed early partly because noise was running in the variant's favor that week, so the lift quoted afterward sits at the optimistic end of what the data supports. The probability statement holds. The number reported next to it does not deserve the same trust.

There is a second gap no framework closes. A test stopped on day three has not seen a weekend and has not let a novelty effect decay. Monitor continuously if the tool supports it, and still hold the test to a full business cycle.

Which should you use?

For most teams the answer is whichever their platform implements. At adequate sample sizes the two frameworks land on the same decisions.

Reasons to prefer Bayesian: stakeholders misread p-values, you want to incorporate prior knowledge from previous tests, or you want a framework that tolerates monitoring.

Reasons to prefer frequentist: your organization has an established significance standard, you are running many tests and want strict control of the false positive rate across all of them, or you need results comparable with existing reporting.

Test quality rests on whether the primary metric was declared in advance, whether the sample was adequate, and whether guardrail metrics were watched. The framework decides none of those. A well-run frequentist test beats a sloppy Bayesian one every time.

Related terms

Statistical significance · Sequential testing · Split testing · Guardrail metrics · Novelty effect

Deeper reading: What is structured A/B testing. Service: Conversion Rate Optimization.

FAQ

Is Bayesian A/B testing more accurate?

Neither framework is more accurate. They answer different questions and, given adequate data, lead to the same decisions. Bayesian output is easier for non-statisticians to interpret correctly, which reduces a real source of error in practice.

What is a prior in Bayesian testing?

An assumption about plausible values before seeing data. Most tools apply a weak default that has little influence at reasonable sample sizes. With very small samples the prior can dominate, so it is worth knowing which one your platform uses.

What probability threshold should you use?

95% probability to beat control is the common default, matching the frequentist convention. The threshold should reflect the cost of being wrong, and it should be paired with a minimum effect size worth shipping.

Can you peek at Bayesian results?

Monitoring is valid in a way it is not for fixed-horizon frequentist tests, because the posterior is a fair summary of the evidence at any moment. Acting on the first favorable reading is the part to avoid, since it overstates the effect and skips the calendar effects a full business cycle covers.