Put an answer layer on the warehouse you already paid for (L4)

The steps

1. 01

   Collect the 20 questions leadership actually asks, verbatim

   Tool: Slack

   Copy the real wording, including the sloppy phrasings. "How's pipeline looking" and "what's our coverage" are two different questions people think are one. Rank by frequency, then cut to the 10 that recur most. Ten is the scope for phase one. • Owner: RevOps • Tool options: search your Slack or Teams analytics channels for 90 days, plus the last 10 board and QBR decks • Pitfall or what breaks: cleaning up sloppy phrasing before recording it, which hides the real ambiguity people have. • Definition of done: 10 questions are written down in the asker's own words, with a count on each.

2. 02

   Freeze the definition of every metric those 10 questions touch

   Tool: dbt Core

   For each metric write four things: the plain-English definition, the exact SQL logic, what is deliberately excluded, and who owns the definition. Pipeline coverage example: coverage of what quota, which stages count, does it include renewals, what is the snapshot date. Get one written answer per question and have the CRO sign it. This is where the project stalls, and it should; every unresolved definition becomes a wrong answer with a confident tone attached. • Owner: RevOps plus the analytics lead, signed off by the CRO • Tool options: a definitions document, then dbt or your semantic layer • Pitfall or what breaks: rushing past unresolved definitions to keep the project moving, which becomes wrong answers with a confident tone. • Definition of done: 10 metrics have signed definitions, and two analysts computing the same metric independently land on the same number.

3. 03

   Implement the definitions in the semantic layer, not in prompts

   Tool: dbt Semantic Layer

   Each frozen metric becomes a governed object with its dimensions and time grains declared. Put the definition text in the object's description field, because that description is what the model reads when it decides which metric to use. A definition living only in a prompt is a definition that drifts the first time somebody edits the prompt. • Owner: Analytics engineer • Tool options: dbt Semantic Layer, Cube, LookML, or your warehouse's native metric layer • Pitfall or what breaks: keeping the definition only in the prompt, so it drifts the first time someone edits the prompt. • Definition of done: all 10 metrics resolve through the semantic layer and return numbers matching the certified dashboard exactly.

4. 04

   Connect the assistant to the semantic layer only, never to raw tables

   Tool: Snowflake Cortex Analyst

   The assistant gets access to the metric layer and nothing beneath it. No raw schema access, no ad-hoc SQL generation against source tables. Enforce row-level security through Okta or Entra groups so a regional director sees their region and no more; the assistant must inherit the asker's permissions, not the service account's. Require every answer to return the number, the metric name, the filters applied, and the as-of timestamp; an answer without those four things is unusable in a meeting because the first question will be "as of when." • Owner: Analytics engineer plus IT • Tool options: Snowflake Cortex Analyst, Databricks Genie, Tableau Pulse, Power BI Copilot, or ChatGPT and Claude via connector or MCP • Pitfall or what breaks: giving the model raw table access because it was faster; within a month two executives quote different revenue numbers from the same assistant. • Definition of done: a permission test proves two users with different roles get correctly different answers to the same question, and every response carries all four elements.

5. 05

   Build a 30-question eval set and gate launch on 95%

   Tool: Google Sheets

   Build 30 questions with known-correct answers: your 10 real questions, 10 rephrasings of them, and 10 that should fail, meaning questions outside the metric layer's scope where the correct behavior is a refusal. Track two rates separately: correct answers, and correct refusals. Do not launch under 95% on both. Re-run the whole set on every semantic-layer change and every model version change, because providers update silently and your accuracy can move without anyone touching your configuration. • Owner: Analytics lead • Tool options: a spreadsheet of question-and-expected-answer pairs, run on a schedule • Pitfall or what breaks: launching under 95% on either correct answers or correct refusals. • Definition of done: the eval set passes at 95%+ twice in a row, a week apart, with no changes in between.

6. 06

   Launch in Slack or Teams with a visible scope boundary

   Tool: Slack

   Publish the list of 10 questions it can answer, in the channel, pinned. Tell people plainly what is out of scope and that out-of-scope questions get a refusal rather than a guess, on purpose. Add a thumbs-down on every answer, and route those to the analytics lead's queue. Add one question per month, only after it clears the eval set; slow expansion is the design, not a limitation. • Owner: RevOps • Tool options: the assistant's Slack or Teams app • Pitfall or what breaks: scope creep to 40 metrics, which drops precision, loses trust, and sends everyone quietly back to Slack-ing the analyst. • Definition of done: 50 real questions have been asked by non-analysts, the thumbs-down rate is under 10%, and one new metric has been added through the eval gate.

7. 07

   Review accuracy monthly and retire what nobody asks

   Tool: Looker

   Monthly, review eval pass rate, thumbs-down themes, and question volume per metric. Metrics nobody asks about get retired so the scope stays tight. Precision falls as scope grows, and a tight scope people trust beats a broad one they check by hand. • Owner: Analytics lead plus RevOps • Tool options: the eval history plus query logs • Pitfall or what breaks: letting a dbt change silently break an answer that nobody notices until a board meeting because no one owns definitions. • Definition of done: two monthly reviews are complete and at least one metric has been added or retired based on real usage.

8. 08

   Instrument the result

   Track analyst hours reclaimed per week, alongside eval pass rate. The first number justifies it, the second keeps it honest. • Where it breaks: the model gets raw table access because it was faster and executives quote different numbers within a month; scope creep to 40 metrics drops precision and trust; or nobody owns definitions so a dbt change silently breaks an answer until a board meeting. • Visual guidance: a four-layer stack: raw tables, transformation, semantic layer, assistant, with a lock icon on the semantic layer and a crossed-out arrow from the assistant straight down to raw tables. That crossed-out arrow is the whole playbook.

Tools in this playbook

• Slack
• dbt Core
• dbt Semantic Layer
• Snowflake Cortex Analyst
• Google Sheets
• Looker

Next playbooks

• Voice-of-Customer → Product & Enablement Loop (L2)
• Zapier Agents for sales ops chores (L2)
• Agentic Nightly CRM Hygiene + Ownership Sync (L3)
• AI-driven price-quote acceleration with CPQ (L3)