Geo Holdouts on a Small Budget: Incrementality Without the Enterprise Tool
The agency report says Meta drove 400,000 dollars last month. The bank account says revenue is flat. Both can be true at the same time, because the platform counts the sales it touched and the store counts the sales that happened, and those are different numbers. The question that settles it is incrementality: how much of that revenue would have happened anyway. The tool that answers it is a geo holdout, and it fits a budget far smaller than the vendors admit.
The question incrementality answers
Attribution answers who touched the sale. Incrementality answers whether the sale needed the touch. A shopper who was going to buy anyway, and happened to see an ad on the way, is not incremental revenue. A platform that reports that sale as its own is not lying; it is answering a different question than the one the store needs. The store needs to know what happens to total revenue when the channel is turned off, and the only way to know is to turn it off somewhere and watch.
The holdout is the experiment. Pick two regions that behave alike, pause the channel in one, keep it running in the other, and compare total revenue, not the channel's own number. The difference is the incremental revenue. Everything else is a model, and models are what got the store into this argument in the first place.
The holdout design that fits a small store
Pick regions, not cities, because cities are too small and too noisy. Two states, or two groups of states, with similar revenue, similar seasonality, and similar traffic mix. Match them on the last eight weeks of total revenue, not on the channel's own number, because the channel's number is the thing being tested. Run the holdout for two weeks, minimum, because one week is a weather report and two weeks is a pattern. Pause the channel completely in the holdout region: no spend, no ads, no retargeting. A partial pause is a partial answer.
The sample problem is real and it has a simple answer. A store doing 50 orders a day cannot detect a 5 percent lift, and should not try. The holdout is for the big question, not the small one: is this channel driving 20 percent of revenue or 2 percent. A two-week holdout on a channel that claims a fifth of revenue will show it, or it will not, and either answer is worth more than another month of platform reporting.
Reading the result without fooling yourself
Score the holdout on total revenue in the region, not on the channel's reported conversions. If total revenue in the holdout region holds steady while the channel is off, the channel was taking credit for sales that would have happened anyway, and the budget moves. If total revenue drops by the amount the channel claimed, the channel was earning its keep, and the budget stays. The mistake is scoring the holdout on the channel's own dashboard, which will show zero by construction, because the channel was off. The dashboard cannot tell you what you need to know. The bank account can. For the platform mismatch that makes this necessary, see why Meta and GA4 never match, and for the promo-code version of the same problem, how promo codes steal credit from organic search.
Frequently asked questions
What is incrementality, in one sentence?
It is the revenue that would not have happened without the channel. Attribution counts the sales a channel touched. Incrementality counts the sales that needed the channel.
How long should a geo holdout run?
Two weeks minimum. One week is noise from weather, paydays, and a single big order. Two weeks is a pattern, and it is short enough that the cost of the test stays small.
Why regions instead of cities?
Cities are too small and too noisy for a small store to read. Two states, or two groups of states, with similar revenue and seasonality, give a signal big enough to trust.
What number do I score the holdout on?
Total revenue in the region, not the channel's reported conversions. The channel's dashboard will show zero by construction, because the channel was off. The bank account is the score.