Reading a Lift Test When the Conversion Counts Are Small
Most store lift tests run on far fewer conversions than the write-up admits. Reading results honestly when the counts are small, and deciding before the test starts what will count as an answer.
A geographic or audience lift test on a mid sized store might produce a few hundred conversions per cell, sometimes fewer. At those counts, a reported lift of twelve percent can sit comfortably inside the range that pure chance would produce, and the same data can be described as promising or as nothing, depending on who is presenting it and what they wanted the answer to be.
This is not a reason to avoid lift testing. On the contrary, a rough causal read beats a confident platform attribution number that was never causal at all. It is a reason to set the test up so its answer, whatever it is, means something, and to report the uncertainty in the same sentence as the lift.
Decide the threshold before you look
Write down, before the test starts, the minimum lift that would change a decision, the test duration, and the spend split. That pre commitment is what stops a noisy result being narrated into a win. If a five percent lift would not change the budget anyway, a test tuned to detect thirty percent is fine, and everyone should know that is the resolution they are getting.
Duration matters in a specific way: run whole weeks. A test that starts on a Wednesday and ends on a Tuesday has not balanced weekdays against weekends in the same way for both cells unless the split is geographic and simultaneous. Concurrent cells save you here, which is one reason geo splits read cleaner than before and after comparisons at small counts.
Report the interval, not just the point
A lift figure without a range is a claim the data rarely supports. At small counts the honest output sounds like: lift estimated at eight percent, with a range from minus six to plus twenty four. That sentence feels weaker in a meeting and it is worth ten times more, because it stops a budget being moved on a difference that may not exist.
Also report the raw counts the estimate rests on. Conversions per cell, spend per cell, and the conversion rates themselves. Readers can sanity check a test they can see. A single percentage with no counts behind it asks for trust, and this corner of marketing has spent its trust.
What to do with an inconclusive result
Inconclusive is a result. It tells you the effect, if any, is smaller than your test could detect at that spend, which rules out the large effect someone may have been claiming. Record it that way, with the detectable threshold stated, and do not run the same underpowered test again hoping for a cleaner story. Pool a longer window, increase the split contrast, or accept a different measurement method.
The failure mode to avoid is quietly extending the test until the number crosses into significance and then stopping. That practice manufactures findings. If the test needs to run longer, decide that on the counts and the calendar, document it, and keep the original end date visible in the write-up. Your future self, reading this in a budget meeting, is the audience for that honesty.
Related reading
If this topic is useful, these earlier guides go deeper on connected questions: View Through Conversions: A Sceptical Read Before You Pay for Them, Geographic Holdouts Without Fooling Yourself: A Practical Setup, and Reading Post Purchase Surveys Without Fooling Yourself.
Frequently asked questions
How many conversions does a lift test need?
It depends on the size of lift you need to detect and your baseline conversion rate. Detecting a small lift takes far more conversions than detecting a large one. Work it out before the test from your baseline, rather than discovering afterwards that the test could only see huge effects.
Why report a range instead of a single lift number?
Because at small counts the single number is unstable. A range shows what the data can and cannot rule out, and stops budgets moving on a difference that could easily be chance.
Is an inconclusive test a waste of money?
No, if it was set up honestly. It rules out effects larger than the detectable threshold, which is real information, especially when a channel was claiming a large effect. It is only wasted if the same underpowered test gets repeated.
Why not just run the test until it becomes significant?
Stopping when a result crosses a threshold manufactures false findings, because random variation will cross thresholds eventually. Set duration and decision rules in advance, and document any extension and the reason for it.