What is segmentation analysis?

Which segments are worth analyzing?

Ranked by how often they reveal something that changes a decision.

Device. The most reliable source of large differences. Mobile funnels leak at forms and checkouts in ways desktop funnels do not, and a blended figure hides it.

Traffic source. Paid, organic, direct, and referral arrive with different intent and different expectations. Paid traffic arriving on a specific promise behaves nothing like organic traffic arriving on a question.

New versus returning. Different knowledge and different expectations, and the split that separates a genuine improvement from a novelty effect among new visitors, or from returning visitors reacting to the change itself rather than to its merits.

Landing page. The segment that most often explains a site-wide rate change, since a shift in which pages receive traffic moves the blended number without any page having changed.

Geography, where language, payment methods, or regulation differ.

Firmographics for B2B, where company size or industry is known. Usually the highest-value segmentation and the hardest to obtain, since it requires enrichment or self-reported data.

Why are post-hoc segments dangerous?

Because with enough segments, something always looks significant.

A test showing no overall effect, examined across ten segments at 95% confidence each, will produce roughly one apparently significant segment by chance alone. Reporting that segment as a finding is not analysis, it is a search.

The distinction that matters:

Pre-registered segments are declared before the test, with a stated reason. They count as evidence.

Exploratory segments are found afterward. They generate hypotheses and prove nothing, and the correct next step is a new test designed to check the specific claim.

Both are legitimate. Presenting the second as the first is not. The practical rule is to label every segment finding as confirmatory or exploratory when reporting it, which forces the distinction into the open.

Key takeaway: a segment discovered after the fact is a hypothesis, not a result.

How do you act on a segment difference?

1. Check sample size within the segment. Splitting divides the sample, so a segment at a quarter of your traffic needs four times the duration for equivalent confidence.
2. Confirm the difference is behavioral, not compositional. Mobile visitors may convert worse because they are mobile, or because mobile traffic skews toward a lower-intent source. Cross-segment by both before concluding.
3. Size the opportunity. A segment converting badly at 3% of traffic is worth less attention than one converting slightly badly at 60%.
4. Decide between fixing and separating. Either repair the experience for the underperforming segment, or serve it differently through targeting.

Related terms

Cohort analysis · Funnel analysis · Statistical significance · Mobile conversion rate · Guardrail metrics

Deeper reading: What is structured A/B testing. Service: Conversion Rate Optimization.

FAQ

How many segments should you analyze?

For confirmatory analysis, two or three declared in advance. Exploratory work can look at more, provided the findings are labeled as hypotheses rather than results.

When should you use a cohort instead of a segment?

When the question involves elapsed time since a starting point. A segment tells you who is converting worse right now. A cohort tells you whether the group that arrived in March is still behaving the way March groups used to. On sales cycles longer than a single session, the cohort view catches conversions the segment view has already written off.

Why does segmenting make my results inconclusive?

Because each segment carries a fraction of the sample, and smaller samples support less certainty. This is the analysis being honest rather than a problem to work around. If a segment matters enough to make a decision on, power the test for that segment from the start instead of splitting a sample sized for the whole.