The agent control plane (L4)

The steps

1. 01

   Build a register of every AI call in the company

   Tool: Okta

   Find every workflow touching customers or revenue data, including the one an enthusiastic AE built in Zapier six months ago. Search expenses for AI vendor names, pull the SSO app list, and ask two questions in a form: what AI workflows have you built, and what data do they touch. Record per workflow: owner, model and version, monthly cost, data it touches, whether a human reviews output, and whether it writes to any system. No row leaves the owner column blank, an unowned workflow gets shut off, not documented. • Owner: RevOps plus IT • Tool options: finance expense reports, the Okta or Entra app inventory, and a two-question survey • Pitfall: leaving an owner column blank instead of shutting the workflow off • Definition of done: the register exists with a named owner on every row and no "unknown" anywhere in it

2. 02

   Route everything through one gateway, highest-spend first

   Tool: LiteLLM

   No application calls a model provider directly. One gateway means one place for keys, rate limits, logs, and model swaps. Migrate the highest-spend workflow first so the value shows immediately, then work down the register. Rotate and revoke every direct provider key as you migrate, otherwise the old path stays quietly open and your logs are incomplete without anyone knowing. • Owner: GTM engineer plus platform engineering • Tool options: LiteLLM, Portkey, Cloudflare AI Gateway, or Azure AI Foundry / Vertex AI / AWS Bedrock • Pitfall: leaving old direct provider keys live after migration, leaving logs incomplete • Definition of done: 80%+ of AI spend flows through the gateway, per-workflow spend is visible, and direct keys are revoked

3. 03

   Log the full trace, including inputs, with retention rules agreed first

   Tool: Langfuse

   Capture prompt, retrieved context, model and version, output, latency, cost, and the human decision that followed. Trace-level logging is what lets you answer "why did it say that" three weeks later, which you will be asked. Settle retention length and PII redaction with legal before you switch it on, not after. Then prove it works by reconstructing one historical incident entirely from logs. • Owner: GTM engineer • Tool options: Langfuse, Helicone, LangSmith or the gateway's native tracing • Pitfall: switching on logging before retention length and PII redaction are agreed with legal • Definition of done: every gatewayed workflow produces a queryable trace and one past incident has been reconstructed from logs as a test

4. 04

   Set a hard budget per workflow with an automatic kill

   Tool: Slack

   Monthly cap per workflow, alert at 70%, automatic pause at 100% that pages the named owner. Then test the pause in staging, because an untested kill switch is not a kill switch. A retry loop in a broken agent will spend a quarter's budget over a weekend and it will do it on a Saturday night. The cap costs nothing until the night it saves the year. • Owner: RevOps plus finance • Tool options: gateway spend limits, alerting into Slack or Teams • Pitfall: shipping a kill switch that has never been tested • Definition of done: a cap exists on every workflow in the register and the pause has been tested successfully

5. 05

   Attach a 20-case eval set to every production workflow

   Tool: Braintrust

   Minimum 20 cases per workflow with known-good outputs, including at least five that should produce a refusal. Run the set on every prompt change, every model version change, and once a week regardless. Weekly matters because providers update models silently. Your accuracy can drop without anyone on your team touching anything, and the eval history is the only way you find out before a customer does. • Owner: each workflow's named owner • Tool options: Braintrust, LangSmith, Promptfoo, or a spreadsheet plus a scheduled run • Pitfall: building an eval set once and never running it again • Definition of done: every production workflow has an eval set and at least four runs of pass-rate history

6. 06

   Review it monthly in the revenue meeting and kill something

   Tool: Manual

   Per workflow show four things: cost, volume, eval pass rate, and the revenue metric it serves. Any workflow that cannot name a metric gets turned off. A tool has to reduce cost, raise revenue, raise margin, cut risk, or create raving fans. Point it at one of those or shut it down. Killing one thing in the first two reviews is what proves this is a control plane and not a status report. • Owner: CRO • Tool options: one page • Pitfall: letting the monthly review become a status report with nothing actually killed • Definition of done: the review has run twice and at least one workflow has been deliberately killed

7. 07

   Instrument the result

   Instrument: cost per successful outcome, per workflow. Not cost per token, not total spend. Tokens are an input; outcomes are what you are buying. • Where it breaks: it reads like infrastructure with no revenue story, so it gets deferred until a surprise invoice or a customer-facing wrong answer forces it through in a panic. Second failure: the gateway goes in but direct keys stay live, so the logs are partial and give false confidence. Third: evals get built once and never run again, which is the same as not having them. • Visual guidance: applications on the left, one gateway in the middle, model providers on the right, with logging, budget and eval boxes hanging off the gateway. Then draw the dotted "before" arrows going straight from apps to providers. The contrast is the entire pitch.

Tools in this playbook

• Okta
• LiteLLM
• Langfuse
• Slack
• Braintrust
• Manual

Next playbooks

• Voice-of-Customer → Product & Enablement Loop (L2)
• Zapier Agents for sales ops chores (L2)
• Agentic Nightly CRM Hygiene + Ownership Sync (L3)
• AI-driven price-quote acceleration with CPQ (L3)