Geo Holdouts for Paid Social: Reading a Messy Result Honestly
A geo holdout sounds clean and arrives messy: uneven regions, a sale in one market, spillover from national delivery. How to set one up and how to report what it actually showed.
The appeal of a geo holdout is that it promises a straight answer. Turn paid social off in a set of regions, keep it on in matched regions, compare revenue, and learn what the spending causes rather than what it correlates with. Then the test runs, one region has a local event and a sales spike, another is hit by a delivery problem, and the straight answer arrives with a lot of explaining attached.
That mess is not a reason to avoid holdouts. It is a reason to design them so the mess is visible and bounded, and to report the result with its conditions attached rather than as a single lift percentage that travels around the business losing its caveats at every step.
Match on the history, then lock the list
Choose regions on pre test behaviour, not on convenience. Match pairs or groups of regions on revenue per capita for the category, prior paid social exposure, urban and rural mix, and fulfilment lead times, because a region with two day delivery and a region with five day delivery will not respond the same way to the same ad. Use the previous eight to twelve weeks as the matching base, and check the match by plotting the paired regions against each other before the test starts.
Lock the region list and the test dates before launch, and write down the exclusions: no national promotions that cannot be regionalised, no other channel switching off in the same regions at the same time. Every exception discovered afterwards weakens the comparison, so the list of things that did not stay constant is part of the method, not an embarrassment to be left out of the write up.
Spillover and the border problem
People live in one region and work, travel, and get targeted in another. Platform location targeting is based on a mix of home, device, and activity signals, none of them exact, so some of the holdout population will see ads and some of the exposed population will not. That contamination pushes the measured difference toward zero, which means a holdout usually understates a real effect rather than inventing one. Say so in the result.
Border areas are where this concentrates. If two matched cities sit either side of a state line with a shared media market, they are a poor pair however well their revenue histories match. Prefer regions with some separation, accept fewer and larger regions over many small ones, and resist the urge to add a region late because early numbers look interesting. Adding regions mid test is how a clean design becomes a story.
Report a range and a decision
The honest output is a range with a method note: lift estimated between these bounds, over this period, in these regions, with these known disturbances. A result that says paid social lift was 4 to 9 percent, with one test region excluded for a local event documented here, is more useful than a point estimate of 6.2 percent that nobody can defend in a budget meeting.
Tie the range to the decision it was meant to inform. If even the low end of the range clears the return the business requires, the decision is straightforward. If the whole range sits below it, that is also a finding. The awkward middle, where the range straddles the threshold, usually means run it longer or larger next time rather than arguing harder about this run.
Related reading: Glossary, About, Incrementality Tests for Branded Search: Is the Click Bought or Free.
Frequently asked questions
What is a geo holdout for paid social?
A test where advertising is withheld in a set of matched regions and continued in comparable regions, so revenue differences can be used to estimate what the advertising causes rather than what it merely coincides with.
How should test and holdout regions be matched?
On pre test revenue behaviour, prior advertising exposure, population and urban mix, and fulfilment lead times, using several weeks of history. The match should be checked before the test starts, then the region list locked.
Does spillover invalidate a geo holdout?
No, but it biases the measured effect downward, because some holdout customers will still see ads. It should be acknowledged in the result, and widely separated regions reduce it.
How should a holdout result be reported?
As a range with a method note covering the period, the regions, and any disturbances or exclusions, tied to the decision the test was meant to inform. A single precise lift figure without those conditions misrepresents the evidence.