{{ p.title }}

A product page has a few hundred milliseconds to become useful, and the recommendation module shares that time with images, prices, stock checks and everything else on the page. A decision that arrives late is either dropped in favor of a default or shifts the layout after the shopper has started reading. In both cases the quality of the decision stops mattering, because the shopper never sees it as intended. We hold our decision path to a budget of under 20 milliseconds, and that number has shaped the system more than any choice of model.

Why a budget instead of a target

A latency target describes what a system usually achieves. A budget is a constraint every component has to fit inside, set before the components are designed. The difference shows up in the slowest requests. A system tuned for a good average can still be slow for one request in a hundred, and on a busy storefront one in a hundred is thousands of shoppers a day. Designing to a budget means paying attention to the 95th and 99th percentile response times, since those are the requests where the page stops waiting.

A budget also forces decisions about what belongs in the request path at all. Anything that cannot complete inside it has to move elsewhere: computed ahead of time, cached, or left out.

Where the time goes

A ranking request in a decision system typically passes through the same stages on every call. Their relative cost varies with the catalog and the surface, but the stages and the levers that control them are consistent:

Two design principles usually follow. Expensive work moves out of the request: product representations, catalog indexes and lookup tables are prepared in advance, so a request reads precomputed data instead of producing it. And the number of products scored per request stays bounded. Scoring an entire catalog of tens of thousands of products on every request will not fit a 20-millisecond budget; scoring a well-chosen shortlist will.

Why the reason is built inside the budget

Explanations are often added as a second step, sometimes as a call to a language model after the ranking returns. That step alone can take longer than our entire budget. We assemble the reason from the scoring output itself, so it costs little more than a lookup and adds no network round trip. The architectural case for building reasons this way is in our article on why explainable AI is an engineering problem. The latency case is simpler: an explanation that misses the page is as useless as a late recommendation.

When the budget is missed

No system meets its budget on every request. Networks stall, caches miss, dependencies slow down. What matters is what the page receives when that happens. The calling page should set a timeout based on its own rendering needs and define a fallback for when it passes: a sensible default ranking for the surface, so the shopper sees reasonable products instead of an empty slot or a late shift in layout.

Fallbacks should be visible in measurement. If a surface quietly serves defaults for a meaningful share of requests, its results understate the system and overstate the stability of the page. Logging which requests fell back, and tracking that rate alongside latency percentiles, keeps both numbers honest.

Measuring latency where it is felt

Latency figures are easy to flatter. A median measured inside the service leaves out network time and the slow tail. Our budget is set for the decision path. The page also pays for network time between the shopper, the storefront and the decision service, and that time belongs in the page’s own budget. The numbers that describe a shopper’s experience are measured from the calling page, reported as percentiles and broken down by surface and region.

Trade-offs

A strict budget limits model choice. A larger model that scores better in an offline test may not fit, and its gain has to be weighed against the cost of serving it late or not at all. Precomputation introduces staleness: a product representation prepared overnight will not reflect a price changed this morning unless the pipeline updates it. Bounding the number of candidates means some products are never scored for a given request, so the quality of the shortlist caps the quality of the final ranking.

These costs are acceptable for a decision that has to appear while the shopper is looking. They would not be for an overnight batch job, such as choosing products for a weekly email, where a slower and larger model is often the better choice. The right budget depends on where the decision is shown.

The engines that make up the decision path, and the order they run in, are described on our Engines page.

See the engines in the decision path