Attribution PlaybookMeasurement that survives the browser

Incrementality Tests and Attribution Reports Disagree on Purpose

The attribution report credits the channel with a strong return, the lift test says the same channel added almost nothing, and the meeting treats one of them as broken. Neither is. They were never measuring the same thing.

Credit division versus cause

Attribution is an accounting system: it takes conversions that occurred and divides credit among the touches that preceded them, by a rule, last click, data driven, or otherwise. Every rule divides the same pie, so attribution can never conclude that the pie would have been the same size with no marketing at all. Incrementality is an experiment: hold out a comparable group, run the channel for one group and not the other, and compare outcomes. Its question is causal, and its answer is routinely smaller, because some credited conversions were going to happen anyway, through habit, search intent, or another channel.

Brand search is the standing example. Attribution hands it superb returns because it sits last before purchase. A holdout usually shows a large share of those buyers arriving through organic links or direct visits when the ads are off, which does not make brand search worthless, since competitors bid in that space, but reprices it from a growth engine to a defensive cost. Both descriptions come from the same spend. Only one of them should set the budget.

Why the experiment is not the villain

Lift tests have real limits and honest practitioners name them: they need volume to detect small effects, they run for a window that may miss long consideration cycles, and geo based holdouts leak when customers cross the boundary. A test that cannot detect an effect has not proven there is none. What a well powered test does prove, when it finds near zero lift under heavy attributed credit, is that the attribution rule is describing position in the journey, not contribution to it.

The eCommerceFuel merchant reporting tradition makes the same point from the operator's chair: the channels that look best in platform dashboards are often the ones closest to the checkout, while the spend that created the demand gets divided among touches the platforms cannot see. A blended view, revenue against total marketing cost, is crude and stays honest precisely because it refuses to divide credit it cannot verify.

Running both systems without letting them fight

Assign each system its question in writing. Attribution, used consistently over time, is a diagnostic: it shows journey shape, assists, and where measurement breaks. Incrementality is the budget instrument: periodic, focused on the largest spends, and refreshed when the mix changes materially. When the two disagree, the experiment wins for money decisions and the attribution report wins for debugging tracking, and a disagreement is logged as information, not treated as an error to be averaged away.

Cadence matters more than precision. Two or three solid lift reads a year on the biggest channels will correct more budget waste than a perfect attribution model reviewed weekly. The goal is not a number everyone trusts. The goal is a business that knows which of its numbers is allowed to move money.

Related reading on this site: Glossary, About, Double Counted Purchases: The event_id Mismatch Behind Inflated Meta Numbers.

Write down, for your largest channel, which system is allowed to change its budget. If the answer is whichever report someone opened last, that is the problem to fix first.

Frequently asked questions

Why does attribution credit conversions an incrementality test says were not caused?

Attribution divides credit for conversions that happened among prior touches. It cannot detect that some buyers would have purchased anyway. A holdout test measures that counterfactual directly, so its estimate of a channel's effect is usually smaller.

Does a weak lift result mean a channel is useless?

Not always. Brand search, for example, often shows modest lift because buyers find another route, yet it defends against competitors bidding on your name. The test reprices the channel's job. It does not automatically end it.

What are the limits of incrementality testing?

Tests need enough volume to detect effects, run over limited windows that can miss long buying cycles, and geo holdouts can leak. An underpowered test that finds nothing has not proven nothing exists. Power and duration belong in the read.

Which should set the budget, attribution or incrementality?

Incrementality, for the channels large enough to test. Attribution remains useful as a consistent diagnostic of journey shape and tracking health. The failure mode is letting platform attribution, which grades the platform's own homework, move money on its own.