How to Evaluate an AI Vendor’s Claims

Most AI vendors can show a number that looks impressive: a lift percentage, an accuracy score, a wall of logos. Few of those numbers survive a handful of follow-up questions. The useful skill for a buyer is knowing which questions turn a number into evidence, and which answers should end the conversation. What follows is the set of questions we would want a buyer to ask us.

Questions about results

What was the comparison? A randomized holdout over the same period is evidence. A before-and-after comparison mixes the system’s effect with everything else that changed.
Which revenue is being reported? Influenced, attributed and incremental revenue can differ widely, and only incremental revenue measures what the system caused. We explain the difference in why before-and-after comparisons can’t prove an AI feature works.
How large and how long was the test, and how uncertain is the result? A lift reported without a sample size or an interval cannot be judged.
Was the metric chosen before the test? A metric picked afterwards, from several candidates, inflates the result.
Who measured it? Results computed in the customer’s own analytics carry more weight than results computed by the vendor.

Questions about how the system works

Where do explanations come from? An explanation generated after the ranking can contradict the decision it describes, as we argue in why explainable AI is an engineering problem.
What happens with a visitor the system has never seen? Many systems fall back to bestsellers on a first visit.
How fast is a decision at the 95th and 99th percentile, measured from the page? Averages measured inside the service hide the requests that miss the page.
How are business rules applied, and can the team see which rule removed a product?

Questions about data

The data questions are concrete: which fields the integration accepts, who holds the key that could re-identify a shopper, where data is stored, how quickly it is deleted on request, and what the system is prevented from inferring. Certifications deserve the same specificity. Ask for the report and its scope rather than a badge, and ask for dates where an audit is still in progress. Our own status is published in the Trust center: SOC 2 Type I and Type II are in progress and expected in the fourth quarter of 2026.

Questions about operations and exit

Ask who is on call when decisions go wrong, and what the uptime commitment is, with credits if it is missed. Ask how pricing scales with traffic, since per-decision pricing can turn a successful feature into a growing cost. And ask what happens at the end of the contract: whether data can be exported and how quickly it is deleted.

Answers that should worry you

Some answers are signals in themselves:

a lift figure with no control group;
model accuracy offered as the headline result;
“proprietary” as the full answer to how explanations are produced;
no uncertainty attached to any number;
case studies that describe outcomes without describing how they were measured.

None of these proves a system is poor. Each means the claim has not yet been tested.

Trade-offs

Rigor has a cost for buyers too. A small pilot may not justify a full evaluation, and a newer vendor will not have years of case studies. In both situations the practical answer is the same: run a short pilot against a holdout, with the metric agreed in advance, and let the result answer the questions that sales material cannot.

Read our answers in the FAQs

The takeaway

Ask how every number was measured, which revenue it counts and how uncertain it is. Ask where explanations come from, how rules are applied, what data the system accepts and keeps, and what happens at exit. When the answers are thin, ask for a pilot with a holdout.

{{ p.title }}