The dashboard was green. The answer was wrong.

Most teams judge an AI assistant by whether it is up and fast. Those checks confirm the tool responded and leave the accuracy of the answer untested. AWS recently published a clear example of the gap, from its own sales tools.

What AWS found

AWS runs an internal set of AI agents that let its sellers and sales leaders ask business questions in plain English. One of them, the Customer Insights Agent, builds a view of each customer. On Oct 16, AWS described what its team found when it tested that agent before launch:

• It reported $410.08 of a customer's three-month revenue on a service as "$410.08 Million."
• It labeled Sales Revenue as Sales Revenue excluding Adjustments.
• It offered an annual recurring revenue estimate that appeared nowhere in the data.
• It presented its own calculations as sourced facts.

Through those runs, monitoring stayed green. In AWS's words, it "told us the system had responded, not whether the answer was right."

AWS caught these errors before launch. The post describes a near miss and reports no customer incident. AWS also spells out the cost had it shipped: a seller who repeats an inflated figure can lose credibility with the customer.

Why this matters outside software

Any team that puts an AI assistant between its people and its numbers faces the same risk. An insurance agent quoting a policy total, a distributor checking a key account's quarter, and a hotel group's sales lead preparing for a renewal all trust the figure on the screen. An uptime chart gives none of them a way to check it.

Quality moves after launch

AWS says the data behind these agents refreshes every week, with no release needed for answers to change. It also describes a pre-production test where a data change, a tool change and a paired prompt update were meant to ship together but landed apart. During that mismatch, the evaluation pass rate fell from 78% to 38%.

AWS also describes a case from production. After an upstream schema change, the agent started citing data from the wrong time window, and in AWS's words "the first signal came through a seller inquiry rather than an automated alert." A seller caught it first. AWS does not report any customer impact from the error.

So an answer that passed before launch needs checking again after each weekly refresh.

What to ask before an AI answer reaches a seller

1. Does each answer name the right account, metric, time period and source?
2. Can the tool show whether a number came from the data or was calculated?
3. Who re-tests after a change to the data, the prompts or the tools?
4. Is each tool tested on its own, with the handoffs between tools tested as well?
5. Do quality checks keep running in production, and who gets alerted when they fail?
6. What result earns this tool the right to stay in the workflow?

AWS's own error list is a useful starting checklist: wrong metric, wrong time period, wrong customer, invented figures, wrong currency or unit, and unrequested calculations.

If your team only uses an off-the-shelf assistant, start here. Before a number from an AI assistant goes into an account plan or a forecast, check the account, the metric, the period and where the figure came from.

The role mix

This shifts the role mix toward judgment work and the people who own answer quality. RevOps and enablement teams define what a correct answer looks like, keep the test questions current, and review failures. Sellers keep the judgment call on what goes in front of a customer.

The counter-view

This is one company's account of its own internal system, published on AWS's own blog alongside AWS products. The errors were found before launch, which is the process working as designed. Several figures in the post's dashboard images are marked illustrative. I'd treat the 78% to 38% drop as one pre-production test, and the list of errors as one agent's findings.

Source

AWS Machine Learning Blog, "Building trust in AI agents: A production-ready evaluation framework with Amazon Bedrock AgentCore," Oct 16, 2026: <https://aws.amazon.com/blogs/machine-learning/building-trust-in-ai-agents-a-production-ready-evaluation-framework-with-amazon-bedrock-agentcore/>