When your units are the whole population: Rethinking experimentation for finite settings

Much of the standard experimentation playbook rests on an assumption that the units in your test are sampled from a much larger population. However, in many business settings, those units are the whole population you actually care about. Recognising this shift opens up a more thoughtful, design-led approach to running experiments. One that fits the realities of how decisions are made in industries like aviation, retail, and beyond.

A different kind of experiment

Much of our work involves helping airline clients run high-stakes experiments where the units are routes and markets, not users or sessions. Think pricing tests across 40 routes or network interventions across 30 markets. Retail store tests and geo-experiments share the same structure: the units are few, heterogeneous, and directly decision-relevant, because they are the actual stores, regions, or routes you operate, not a random sample drawn from a larger pool.

This is not simply a small sample size issue. The more interesting question is this: where does uncertainty actually come from when your units are the whole population you care about?

Rethinking where uncertainty comes from

In large-scale digital experimentation, uncertainty is typically framed as coming from sampling. But when your units are routes, stores, or regions, a more natural source of uncertainty is the assignment itself, meaning which units were allocated to treatment and which to control.

Imagine you randomise treatment across 30 routes. Those 30 routes are your object of interest. The key counter factual is not a hypothetical infinite population of routes. It is what the result would have looked like under other valid assignments of treatment across those same 30 routes.

When your units are routes, stores, or regions, a more natural source of uncertainty is the assignment itself, meaning which units were allocated to treatment and which to control.[SK1.1]

The logic of design-based inference

This is the foundation of design-based inference. In these settings, design and analysis are tightly linked. Your inference should reflect the assignment mechanism you actually used, rather than relying on large-sample approximations simply because the output looks familiar.

It is exactly why randomisation-based methods are so valuable in geo-experiments and other finite-population settings. Instead of defaulting to asymptotic formulas, you can evaluate your observed statistic against the distribution implied by your experiment's actual assignment rule.

The role of heterogeneity

Heterogeneity adds another layer to consider. Routes, stores, and regions are genuinely different from one another, which makes uncertainty harder to characterise. Standard off-the-shelf formulas can be less reliable here than is sometimes assumed.

This does not mean standard tools have no place. They remain useful, and often a sensible starting point. The opportunity is to use them with a clear understanding of what your design actually supports.

Routes, stores, and regions are genuinely different from one another, which makes uncertainty harder to characterise. Standard off-the-shelf formulas can be less reliable here than is sometimes assumed.[SK2.1]

A well-established approach, worth revisiting

Design-based inference is not new. It is well established in econometrics, biostatistics, and experimental design. In business circles, where the conversation is often shaped by large-scale A/B testing, it tends to receive less attention than it deserves in the contexts where it fits best.

When this approach fits best

When your units are a finite set, cities, stores, routes, or enterprise accounts, rather than a sample of millions of users, design-based inference is a natural starting point. It aligns your analysis with the way your experiment was actually run, and gives you a clearer, more honest picture of what your results can tell you.

For teams making high-stakes decisions in finite-population settings, that clarity is worth the shift in perspective. The math isn't harder. The mindset is just different: you're no longer estimating what would happen in a hypothetical population, you're measuring what did happen in the one you have.