Incrementality Tests: How to Run One Without a Data Team

Meta says it drove 18,000 conversions last month. Your bank account says revenue is flat. Both numbers can be true at the same time, because attribution counts the conversions that followed an ad, and incrementality counts the conversions the ad caused. Those are different questions, and only one of them tells you whether to keep spending.

Attribution is not incrementality

Attribution asks which touchpoint gets credit for a conversion that happened. Incrementality asks whether the conversion would have happened without the ad at all. A customer who was going to buy anyway and happened to see an ad first is attributed to the ad and worth nothing incrementally. A campaign that harvests existing demand looks excellent in Ads Manager and adds nothing to revenue. That is why the first incrementality test most teams run is also the most uncomfortable one: it measures the channel they like most.

There are three ways to measure it. Meta Conversion Lift is a user-level randomized controlled trial, the same design as a drug study: Meta splits the eligible audience into a test group that can see ads and a control group held out and never served them. It generally needs on the order of 5,000 conversions in the test group for a reliable read, so it fits always-on campaigns at real budget, and it runs through a Meta representative rather than as a self-serve product. Google offers a Conversion Lift study on the same model. The third way, and the one a small team can run itself, is the geo holdout.

The geo holdout, step by step

Pick one question, the biggest doubt you have. Is Advantage+ incremental, or is prospecting carrying the account? One clear question per test. Then build matched markets: 10 to 15 regions with tightly correlated sales history, weekly-revenue correlation above 0.8, split into test and control so the two groups look nearly identical before you touch anything. Size the holdout at roughly 20 to 30 percent of addressable revenue, and run it for four to eight weeks so you clear at least one full purchase cycle.

The sample size arithmetic is unforgiving, and it decides feasibility before anything else. Detecting a 20 percent relative lift takes about 475 purchases per arm; detecting a 10 percent lift takes about 1,730. Below a few hundred purchases in the measurement window, no split will return an answer you can act on. Freeze your conversion definition before the window opens, because a value convention or deduplication rule that changes mid-flight produces lift that is really instrumentation. Change exactly one thing: Meta ads off in the control region, everything else identical, email sends, promotions, other channels, site changes. And do not look early. Read total store revenue and order count for the test region from your own order records, not platform-reported conversions, because the platform number moves for reasons that have nothing to do with incrementality.

What to do with the answer

Well-designed geo tests hit 80 to 95 percent accuracy in detecting true lift. Sloppy ones tell you nothing, and the fastest way to ruin one is changing budgets mid-test. In one published case, revenue in the paused markets fell by about a third of what the platform said Meta was driving, which meant two-thirds of the attributed conversions would have happened anyway. That is the number that changes a budget, and it is the number attribution can never give you.

Two caveats before you act. In April 2025 Meta added an Incremental Attribution setting inside Ads Manager, which uses randomized control groups to predict which conversions the ads actually caused and optimizes delivery toward incremental outcomes. It is a good optimization signal and a poor final scorecard, because it is still Meta grading its own homework. Use it to point delivery at incremental buyers, then confirm with a holdout you control. And run the test two to four times a year, not every week. Incrementality is a strategic measurement, not a creative test, and the accounts that benefit most are the ones spending above roughly $50,000 a month on Meta, where the test is powered and the answer is worth more than the cost of running it. The last-click problem that makes this necessary is Last-Click Attribution Hides Your Assist Channels.

Frequently asked questions

What is the difference between attribution and incrementality?

Attribution asks which touchpoint gets credit for a conversion that happened. Incrementality asks whether the conversion would have happened without the ad. A campaign can look excellent on attribution and add nothing to revenue.

What is a geo holdout test?

You pick matched geographic markets, keep ads running in some and pause them in others, then compare total revenue, not platform-attributed revenue. It measures business-level impact instead of user-level attribution.

How many purchases does an incrementality test need?

Detecting a 20 percent relative lift takes about 475 purchases per arm, and a 10 percent lift takes about 1,730. Below a few hundred purchases in the window, no split returns an answer you can act on.

Can a small team run an incrementality test?

Yes, with a spreadsheet and discipline. You need 10 to 15 matched markets, a four-to-eight-week window, one frozen conversion definition, and the willingness to give up a little attributed revenue for a few weeks to learn something worth far more.