Twenty Questions That Separate an AI Vendor From an AI Demo

The demo is not the product. The demo is the product's best hour, on the vendor's data, driven by the person who built it. You will never see that hour again.

Diligence is how you find out what the other 999 hours look like. Below are the questions that consistently produce a different answer than the sales conversation, grouped by what they protect. Ask them in writing, and treat a vague answer as an answer.

Questions about the model and the output

1. Which models do you use, and what happens to our workflows when you change them?
2. How do you measure output quality today, and can we see the measurement on accounts like ours?
3. What is your documented failure rate, and what does a typical failure look like?
4. How does the system behave when the input data is missing or contradictory? Does it abstain or guess?
5. Can we see three real outputs you consider bad, and what you did about them?

Question five is the one that separates operators from marketers. A vendor with a real quality program can produce bad outputs instantly and talk about them without flinching. A vendor without one will change the subject.

Questions about your data

1. What exactly do you ingest, at what frequency, and what do you retain after we terminate?
2. Is our data used to train or improve any model that another customer benefits from? Where is that in the contract, not the FAQ?
3. Which subprocessors touch our data, and where do they operate?
4. What happens to derived data, embeddings, and summaries on termination?
5. Can we run in a mode where nothing leaves our tenant, and what capability do we lose in that mode?

The last one matters more each year. Many vendors offer a private mode that silently disables the features you bought. Ask what breaks, in writing.

A vendor who cannot show you a bad output does not measure quality. They measure demos.

Questions about your process

1. Where does this sit in the seller's existing workflow, and what does the rep have to open that they do not open today?
2. What is your median time to first value with a customer of our size and our data condition?
3. Which of your customers turned this off, and why? We will accept anonymized answers.
4. What does implementation actually require from our RevOps team, in hours, by week?
5. How does the system handle our exceptions: partner deals, multi-year renewals, whatever is weird about our motion?

The churn question is rarely answered honestly, but the shape of the dodge is informative. A vendor who says nobody has ever turned it off is either very new or not telling you the truth.

Questions about the commercial structure

1. What is the unit of pricing, and what happens to our bill if usage doubles?
2. What are we contractually entitled to if the quality metric we agree on is not met?
3. What is the exit path: data export format, timeline, and cost?
4. What is on the roadmap that we are being asked to pay for today but cannot use yet?
5. Who owns prompts, configurations, and workflows we build on the platform?

Question sixteen deserves emphasis. If the contract has no quality obligation, you have bought access, not outcomes. That may be fine. It should be a decision, not a discovery.

How to run the evaluation so the answers matter

Send the questions before the second demo, in a document, and require written responses. Score them yourself against what you know about your own environment. Then run a bounded trial on your ugliest segment, not your cleanest, because clean data will make three vendors look identical and ugly data will rank them accurately.

Define what qualified success looks like before the trial starts, and name the person who decides. Trials without a decider become subscriptions by default.

The meta-signal

Across dozens of these evaluations, the strongest predictor of a good outcome is not the model, the funding, or the logo wall. It is how the vendor behaves when you ask a question they cannot answer well. The ones who say we do not measure that yet, and here is what we do measure, tend to ship. The ones who reframe tend to churn.

You are not buying software. You are buying a working relationship with a team that will be wrong sometimes. Evaluate the wrongness.