{{ p.title }}

Teams evaluating a recommendation model usually compare offline scores: precision, recall, a ranking metric computed on last month’s data. A new model that scores higher looks like progress, and sometimes it is. Often the improvement never reaches the business. Conversion stays flat, or rises while returns rise with it. The model got better at predicting, and the decisions it fed did not improve, because predicting and deciding are different problems.

Prediction and decision are different problems

A model predicts something about the world, such as the probability that a shopper clicks or buys a product. A decision chooses an action given that prediction, the alternatives available, the constraints that apply and what the business is trying to achieve. Offline metrics score the prediction against what shoppers did in the past. Decision quality can only be judged by what happens when the decision is made, compared with what would have happened otherwise.

The two are related, and a much worse model rarely makes better decisions. Between them, though, sit several steps where an improvement in prediction can disappear.

Where accuracy gains get lost

• The target is a proxy. Models are often trained to predict clicks because clicks are plentiful. A model that predicts clicks better can promote products that attract attention and get returned. The outcome the business wants is a purchase that is kept, and it arrives later and less often.
• Constraints remove the best predictions. Offline evaluation usually scores an unconstrained ranking. In production, stock, margin floors and exclusions remove products, and a model whose advantage sat in products that cannot be shown gains nothing.
• The model cannot see the context. A shopper buying a gift and the same shopper buying for themselves look alike in historical data. A model trained on that data can be accurate on average and wrong for the occasion.
• The data reflects the old system. Logged behavior only covers products that were shown, in the positions they were shown in. Evaluated on it, a model is rewarded for agreeing with the previous system, which overstates gains for models that imitate it and understates gains for models that would choose differently.
• A person still has to act. A correct recommendation that a shopper does not understand or trust may simply be ignored.

Measuring the decision instead of the model

Offline metrics remain useful during development, for screening candidate models and catching regressions before anything reaches a shopper. They are a weak basis for claims about value. The measure of a decision system is its effect on an outcome the business cares about, against a randomized holdout, over enough time for that outcome to arrive. In practice that means a primary metric such as conversion or order value, read alongside guardrail metrics such as return rate and margin, so a gain in one is not bought with a loss in another. We describe how to design that comparison in why before-and-after comparisons can’t prove an AI feature works.

Design choices that close the gap

Several design choices narrow the distance between a better prediction and a better decision:

• train toward the outcome that matters, such as a purchase that is not returned, rather than the most plentiful signal;
• apply business rules inside ranking, so the model optimizes among products that can actually be shown;
• read the context of the session, so the same person is treated differently on different occasions;
• reserve some exposure for exploration, so the system learns about products and positions the old system never tried;
• attach a reason to each decision, so the person on the other end has grounds to act on it.

Trade-offs

Outcome-based targets are slower and noisier. Returns arrive weeks after a purchase, so a model trained to avoid them learns from older data and less of it. Exploration costs short-term performance, since some decisions are made partly to learn. And context is expensive: capturing occasion and intent well requires careful design of the dimensions for each category.

None of this makes model quality irrelevant. A better model is necessary for a better decision system. It is not sufficient, and the gap between the two is where much of the engineering effort in a real deployment goes.

See how the platform turns predictions into decisions