Geographic Holdouts Without Fooling Yourself: A Practical Setup
Geo tests are the incrementality method most stores can actually run. They fail on selection, spillover, and impatience. How to set one up so the answer means something.
A geographic holdout is simple in outline: pick comparable regions, advertise in some, hold back in others, compare sales. It is the test most mid sized stores can run without platform cooperation, which is its appeal. It is also easy to run badly in ways that guarantee a pleasing answer, usually by choosing the regions after looking at their recent sales.
The discipline is all in the setup. Once the test is running, there is almost nothing legitimate left to adjust.
Choose regions before you look at outcomes
Match regions on the things that drive your sales: population, past sales per head, retail presence, climate where it matters, and media overlap. Broadcast and some digital buying spill across region borders, so prefer regions that are separated in media terms, not just on a map. Commuter belts poison holdouts quietly.
Randomise within matched pairs where you can, and write down the assignment, the metric, and the end date before launch. A primary metric, usually revenue or orders per head against the matched baseline period, plus a minimum effect size worth acting on. Without that, any difference, in either direction, becomes a story someone can tell.
Run it long enough, touch nothing
The test needs to cover your normal purchase cycle, including at least one full weekly pattern and any payday effect your category has, which for most stores means several weeks, not several days. Short tests mostly measure noise and the novelty of a change in spend.
During the run, resist corrections. A promotion that lands in test regions and not control regions, a stockout in one group, a local event: log them, do not quietly rebalance the groups mid test to compensate. Every mid test adjustment is a choice about the answer. If something catastrophic happens, end the test, say why, and restart cleanly later.
Read the result like an operator
Compare against the baseline relationship between the groups, not just the raw test period gap. If test regions historically ran 5 percent ahead, a 5 percent lead in the test is a null result wearing a costume.
Then do the arithmetic in money. A statistically tidy 2 percent lift on a channel may not cover its cost once margin is applied. Report the effect, the uncertainty around it, and the decision it implies, in one short document, including when the decision is to stop spending. A test library full of only positive results is a sign the method, not the marketing, is broken.
Related reading
If this topic is useful, these earlier guides go deeper on connected questions: Reading Post Purchase Surveys Without Fooling Yourself, Attribution When Consent Rates Drop and Modelled Conversions Take Over, and Incrementality Tests for Branded Search: Is the Click Bought or Free.
Frequently asked questions
How long should a geographic holdout run?
Long enough to cover your purchase cycle and a full weekly pattern, usually several weeks. Short runs mostly capture noise and day to day variation.
How do I choose test and control regions?
Match them in advance on sales per head, population, retail presence, and media separation, and write the assignment down before launch. Choosing after seeing recent performance biases the result.
What is spillover and why does it matter?
Spillover is advertising reaching control regions, through broadcast areas, commuting, or imprecise digital targeting. It shrinks the measured difference and can make a working channel look ineffective.
What should I do with a null result?
Report it and act on it. A well run null result is valuable information about where budget is not working, and it should usually move spend.