Planning Incrementality Tests You Can Actually Run After Peak
Peak data is tempting and misleading for testing. How to design post peak incrementality tests with clean holdouts and honest read windows.
Every team leaves peak season with a story: a channel surged, a campaign carried the quarter, a tactic must scale. Peak is the worst data to test those stories on, because demand was abnormal, budgets were abnormal, and everything ran at once. The test belongs after peak, designed while the story is still fresh.
This guide covers choosing the claim to test, building a holdout that survives stakeholder pressure, and picking a read window that reflects how your customers actually buy.
Turn the peak story into one testable claim
Write the claim as a single sentence with a number: this channel causes additional orders beyond what would happen anyway, at roughly this rate. Vague claims like brand search works cannot be tested. A claim with a size attached can be, and can be wrong in a useful way.
Pick the claim with the biggest budget consequence, not the most interesting one. Incrementality work earns its keep by changing an allocation decision. Testing a small channel beautifully while the largest spend goes unquestioned is analysis as decoration.
Holdouts that hold
A geo or audience holdout only works if it stays held. Agree in advance what would justify breaking it, usually a major trading emergency, and who can authorise that. A holdout that leaks every time a sale dips produces data that looks rigorous and means nothing.
Size the holdout for the claim, not for comfort. Too small a control group cannot detect the effect you claimed, and the test concludes no effect when it really concluded nothing. Run the power calculation before launch, with last years post peak baselines, not peak numbers.
Read windows matched to buying cycles
Measure over at least one full buying cycle for your category, longer for considered purchases. A test read at one week on a product people research for three will credit only the fastest buyers and misprice the channel for everyone else.
Pre commit the read date and the decision rule: what result scales spend, what result cuts it, and what result reruns the test. Deciding the rule after seeing the number guarantees the number will be argued into whichever answer was preferred all along.
Report uncertainty as part of the result
Present the estimate with its range, plainly. A lift of four percent, plus or minus three, is a different decision from four percent exactly, and stakeholders deserve the honest version. Teams that hide uncertainty get one confident wrong call and lose trust in testing entirely.
Close with the allocation change the test caused, even if the change is none and the reason is the range. A test library with decisions attached becomes the planning base for next peak, which is where this whole exercise pays back.
Related reading
If this topic is useful, these earlier guides go deeper on connected questions: Server Side Tracking Fixed Consent Loss and Created New Gaps, Setting Attribution Baselines Before Black Friday Chaos, and View-Through vs Click-Through Attribution: What Each Number Is Really Claiming.
Frequently asked questions
Why not test incrementality during peak itself?
Peak demand, budgets, and overlapping campaigns are all abnormal, so effects cannot be separated cleanly. Design the test right after peak, while the claim is fresh, and run it on normal trading where a holdout can actually hold.
What makes a claim testable?
One sentence with a size attached: the channel causes a stated rate of additional orders beyond baseline. Claims without a size cannot be sized for a holdout, and vague claims can be argued either way whatever the data says.
How big should a holdout be?
Big enough to detect the claimed effect at a reasonable confidence level, calculated before launch from normal period baselines. An undersized holdout that finds nothing has not disproved the claim, it has wasted the test window.
What should be decided before results arrive?
The read date, after at least one full buying cycle, and the decision rule for scaling, cutting, or rerunning. Committing the rule in advance stops the result being negotiated into the answer stakeholders wanted anyway.