{{ p.title }}

Conversion went up after the new recommendations launched. That sentence appears in a great many AI case studies, and it proves very little. Traffic mix shifts with the season, campaigns start and stop, prices change, and a site redesign often ships in the same month. A before-and-after comparison attributes all of it to whatever was launched. The reliable way to know what a decision system is worth is to compare shoppers who received its decisions with shoppers who did not, over the same period, chosen at random.

What a holdout measures

A holdout is a randomly assigned group of shoppers who keep the existing experience while everyone else receives the new one. Because assignment is random, both groups face the same season, the same campaigns and the same prices. The difference in their outcomes, on a metric agreed in advance, is the incremental effect of the system: the change that would not have happened without it.

This is the logic of a controlled experiment, and it answers a narrower question than most dashboards do: how much revenue the recommendations added, as opposed to how much revenue passed through them.

Three ways to credit revenue to a decision

The distinction matters because vendors and internal teams often report different numbers under the same label. We separate three:

• Influenced revenue is any revenue from a session in which a shopper saw or clicked a recommendation. It is large and easy to report, and much of it would have happened anyway.
• Attributed revenue assigns credit by a rule, such as the last recommendation clicked before purchase. It is useful for comparing surfaces with each other, but the rule determines the answer.
• Incremental revenue is the difference between the treated group and the holdout. It is the only one of the three that measures cause, and the only one that should justify spending.

A credible results report says which of the three it shows. If it does not say, assume it is the first.

Designing a holdout that can be trusted

Randomization has to happen at the right level. Assigning by shopper, using a pseudonymous identifier, keeps each person’s experience consistent and avoids the contamination that occurs when the same shopper moves between groups from one session to the next. Assigning by page view is simpler but blurs the comparison.

The metric has to be chosen before launch, together with the threshold that counts as success. Choosing a metric after seeing the data, or checking results daily and stopping when they look good, inflates the apparent effect. Both mistakes are common, and writing the plan down first prevents both.

The test also has to run long enough. A pilot needs enough sessions for the difference between groups to stand out from noise, and enough time to cover the weekly cycle of the business. Shoppers react to novelty, too: a new module can draw clicks in its first week that fade by the fourth. For most storefronts, a month at a 50/50 split is the shortest period that produces a result worth acting on, which is how we run pilots.

Why the holdout stays after launch

Many teams treat measurement as a launch gate. They prove the lift once, switch the holdout off and give everyone the new experience. From then on the system is unmeasured, while the catalog changes, models are retrained and shopper behavior shifts with the seasons. The result that justified launch becomes a historical claim.

We keep a 10% holdout permanently after the pilot. It costs the business the effect of the system on one shopper in ten, which is a real cost. In return, the incremental effect is measured every month on current traffic, regressions show up as a narrowing gap instead of a slow decline nobody can explain, and every claim about value rests on a current number.

Reading a result

A holdout result is a range with an uncertainty attached, and it should be reported that way. A small effect with a wide interval is a reason to keep testing before declaring success. It also helps to report guardrail metrics alongside the primary one. If conversion rises while average order value or margin falls, the system may be steering shoppers toward cheaper products, and the headline number hides it.

Breaking results down by device, category or traffic source can show where a system helps and where it does not. Slice the data enough ways, though, and some segment will look impressive by chance. The segments worth reporting are the ones named in the plan before the test began.

Trade-offs

Holdouts have real costs. The withheld group misses whatever the system adds, during the pilot and afterwards. Smaller sites may lack the traffic to read a result in a month, which forces a longer test or a decision made on weaker evidence. Some effects are also hard to capture in a holdout at all: improvements that build up over many visits, or that depend on a shopper seeing consistent decisions across channels, can be understated by a test that measures one surface at a time.

None of these costs make before-and-after comparisons a better option, since those carry the same uncertainty and hide it. They argue for planning the test with care: a surface with enough traffic, a metric that moves within the test window, and a duration agreed before launch.

How we report incremental, attributed and influenced results, and what our monthly review covers, is set out on our Results page.

See how we report results