What It Costs to Run a Decision System in Production

Estimates for an AI feature usually start with the model: how long it takes to build and what it costs to serve. In production, the model is a small part of the work. Sculley and colleagues at Google made the point in a 2015 NeurIPS paper, “Hidden Technical Debt in Machine Learning Systems”, which drew the model code as a small box inside much larger systems for data collection, verification, serving and monitoring. A decision system for a storefront fits that picture, and most of its cost of ownership sits around the box.

Where the cost sits

Component	What it involves	Why it gets underestimated
Data pipelines	Collecting behavioral events, syncing the catalog, preparing inputs	They fail quietly when an upstream system changes
Serving	Low-latency infrastructure, redundancy, data regions	It has to meet the latency budget at peak traffic as well as on average
Evaluation	Holdouts, metric definitions, regular reporting	It continues for as long as the system runs
Monitoring	Latency, fallback rate, input drift, decision mix	Someone has to respond when it alerts
Model updates	Retraining, validation, staged rollout	Each update needs the checks the first launch had
Governance	Business rules, reviews, audit trail, privacy controls	It grows with every surface and every rule
People	On-call engineers, merchandiser time, analysis	It is rarely in the original estimate

Drift is the default

A decision system starts drifting the day it launches. The catalog changes, seasons turn, campaigns shift the traffic mix and shopper behavior moves with them. Inputs that were common become rare and new ones appear. Without monitoring, the first sign of drift is a decline in results that nobody can explain, often months later.

The useful signals are cheap to track: changes in the distribution of inputs, changes in the mix of products and categories being recommended, a rising fallback rate, and the gap between the treated group and the holdout. The last matters most, because it measures what drift costs in outcomes rather than in inputs.

Failure modes that cost the most

Stale catalog data produces recommendations for products that are out of stock or have changed price, which costs trust as well as revenue. Silent pipeline failures, where events stop arriving but nothing errors, leave the system deciding on old information. Fallbacks that begin as rare exceptions can become a meaningful share of traffic, a problem covered in our article on latency budgets. And business rules accumulate until the model has little room left to rank, which we discuss in where business rules belong.

Build, buy or partner

Building in-house means owning every row of that table, including the on-call rotation. It gives the most control and the closest fit, and it makes sense where decisions are central to the business and a team to run them already exists. Buying a platform moves serving and much of the monitoring to the vendor. The business still owns its data quality, its rules and its decisions about what to measure, and those need planning whichever route is chosen.

Pricing structure affects the cost of ownership as well. Pricing per decision makes cost grow with traffic, which is hard to forecast for a feature that succeeds. We do not charge per decision, so costs stay predictable as volume grows.

Whatever the route, four questions clarify the real cost: who is on call when the system misbehaves, who owns the data pipelines, how evaluation runs after launch, and what happens to the data when the contract ends. In our own engagements, data is deleted within 30 days of a request.

Trade-offs

Planning for the full cost can make a project look less attractive than a model-only estimate, and some teams resist it for that reason. The alternative is discovering the cost after launch, when it competes with everything else for attention. A small first deployment measured against a holdout is also a cheaper way to learn what the operating cost will be before committing to it at scale.

See how engagements are priced

The takeaway

Budget for the system around the model: pipelines, serving, evaluation, monitoring, updates, governance and the people who run them. Expect drift from the first day, watch the holdout gap, and settle who is on call and what happens to the data before choosing to build or buy.

{{ p.title }}