--- name: substrate-builder description: Converts a loose business prompt into a substrate, an explicit specification (definitions, thresholds, boundaries, precedence, closed labels, missing-data rules, canonical output) that makes different runs and different LLMs return the same answer, then proves it with an oracle coverage check and a cross-model drift test. Use when someone says their AI "keeps changing its answer," "gives different results every time," wants the "same answer every time," asks to audit a prompt for ambiguity or build a scoring rubric for AI, "Claude and ChatGPT disagree," wants "consistent," "reproducible," or "deterministic" AI output, asks to "build a substrate," "define hot lead / qualified / at risk / high value," or uses AI to score, classify, qualify, tier, route, or decide on leads, accounts, deals, tickets, or investments. --- # Substrate Builder A substrate is the external specification of a task's meaning. The human owns the meaning: every concept, threshold, boundary, precedence rule, and allowed label. The model only applies it. When the meaning is written down completely, independently trained models stop choosing their own interpretation and converge on one output. This skill reconstructs the method from Elin Nguyen's research on interpretation drift (OmnisensAI). It is not an official OmnisensAI tool. ## The standard - A substrate passes when **every run, on every model tested, on every wording variant, returns the same normalized output, and that output equals the oracle's answer.** 10 of 10, per model, per variant. - 9 of 10 is a FAIL. It means a second interpretation is still alive, and you do not control when it appears. - A substrate makes a policy repeatable. It does not make the policy right. The human signs off on the policy before anything is tested. ## Scope check (run first) Substrates fit operations with a closed answer space: classify, score into bands, qualify, tier, route, extract into fixed fields, decide among named options. - **Compound request** ("score this lead and write the follow-up email"): split it. Build a substrate for the decision. Generate prose in a separate call that receives the fixed decision as input. - **Open-ended writing only** (a blog post, a summary for humans): tell the user a substrate does not apply to the prose, and offer to substrate the parts that must be consistent (a category, a sentiment label, a next-step type). - **Agent loops and long contexts:** one substrate per decision step, re-sent at each step. --- ## Phase 1: Reconstruct the task object (REQUIRED) Write down, in five lines, the task the user actually needs: OPERATION: e.g. classify UNIT: e.g. a single lead (not the account) EVIDENCE: e.g. firmographics, dated engagement events ANSWER: e.g. ICP_FIT, INTENT, RECENCY, TEMPERATURE, ROUTE CONSUMER: e.g. routing rule in the CRM Show this to the user. If they correct any line, update it before Phase 2. Different runs of a model build different task objects from the same prompt. This step removes that freedom first. ## Phase 2: Ambiguity census (REQUIRED) STEP 2a: Save the user's prompt to a file and run the lexical audit: python scripts/audit_prompt.py STEP 2b: Read `references/ambiguity-census.md` and complete all 14 classes. The script finds keywords. Missing tie rules, units, and aggregation operators have no keyword; find them by walking the classes. Output a census table: | # | Class | Exact words or absence | Plausible readings | Decision needed | |---|---|---|---|---| Mark every placebo line (be consistent, do not hallucinate, think step by step, role prompts, all caps) for deletion and say what rule replaces it. ## Phase 3: Policy decisions (REQUIRED; human sign-off) For each "decision needed," propose one rule. Label every one **PROPOSED**. For common GTM concepts, start from `references/gtm-ontology-starters.md`. - Base proposals on what the user told you about their business. - If you have no basis, give two options and the consequence of each. Do not pick silently. - Never present a proposed threshold as an industry standard. Ask for confirmation in one message. **Do not compile the substrate until the user confirms or explicitly says to use the proposals as written.** A substrate built on unconfirmed policy is a reproducible guess. ## Phase 4: Compile the substrate Read `references/substrate-grammar.md` (14 writing rules with bad/good pairs). Fill `assets/substrate-template.md` in its section order: Objective, Authoritative Input, Input Conventions, Consistency Rules, Definitions, Decision Rules, Precedence, Routing, Canonical Output (last). Non-negotiables: - Every derived field has an UNKNOWN path; missing data is decision rule 1. - "none on record" (known empty) is different from MISSING (unknown). - Every label set is closed and ends in a fallback. - Every threshold states which side its exact value belongs to. - Arithmetic is done in code and passed in as a fact. - No placebo lines. No role line. No examples with real-looking values inside the substrate. ## Phase 5: Prove the rules are complete (REQUIRED, before any model sees them) Translate the substrate into an oracle: a plain Python function with the same fields, values, and rule order. Start from `assets/oracle_template.py`. Build a CASES grid with, for every threshold, the value below, the value itself, and the value above, plus None and MISSING where they apply. python scripts/coverage_check.py --write-expected expected.txt It checks that every input gets exactly one legal value for every field, that every label is reachable, and that every rule fires. **If it reports FAIL, the substrate has a hole. Fix the substrate, then the oracle, and rerun. Do not proceed to Phase 6 on a FAIL.** If the oracle and the substrate text disagree on any case, the text is ambiguous; rewrite the text. Example: the first draft of the acquisition substrate left 24 of 100 boundary cases with no RISK value (growth 9.9%, margin 9.9%, top three customers 29.9% matched no rule). A model given that draft must invent a risk level. Adding "MEDIUM otherwise" closed it. See `examples/acquisition_substrate.md`. ## Phase 6: Drift test Build the test matrix: - **Runs:** 10 per cell. - **Models:** at least 3, from at least 2 vendors, including every model that will run this in production. Pin exact model IDs. - **Variants:** the substrate plus at least 2 contextual variants that keep every fact and rule but change the wrapper: reorder the Authoritative Input lines; add one neutral sentence before the Objective ("Here is today's record."). The answer must not move. Run it either way: # no API keys: paste each output into runs///runNN.txt python scripts/drift_test.py files runs/ --substrate substrate.md --expected expected.txt --out drift-report.md # with keys in the environment python scripts/drift_test.py api --prompt substrate.md --prompt variant_reordered.md \ --prompt variant_preamble.md --model anthropic: --model openai: \ --model google: --n 10 --save runs/ --expected expected.txt --out drift-report.md API mode sends the substrate text, and nothing else, to the providers named on the command line (endpoints listed in the script header). Files mode makes no network calls. Leave temperature at the provider default. A substrate that converges only at temperature 0 still has open meaning; the sampler is just hiding it. The report separates temporal, spatial, and contextual drift and lists the fields that diverged. Read `references/drift-taxonomy.md` to interpret it. Optional baseline: run the user's original prompt through the same matrix first. The before/after reports are the evidence that the substrate, and not luck, produced the convergence. ## Phase 7: Diagnose and patch On FAIL, open `references/diagnosis-playbook.md`. Take the first diverging field, match its signature, and make **one** change. Bump the version, update the oracle if a rule changed, rerun coverage, rerun the full matrix. Log the minimal pair: text before, text after, result before, result after. Stop and escalate to the user when the divergence reflects a real policy disagreement, or when every new record needs a new rule (the task is more than one decision). ## Phase 8: Release Release criteria, all required: 1. coverage_check.py: PASS 2. drift_test.py: PASS on the full matrix, against expected output 3. 20 real records run through the substrate and spot-checked by the policy owner (the substrate's answers match what the owner would decide) Deliver to the user, in this order: 1. Task object (Phase 1) 2. Census table (Phase 2) 3. Confirmed policy table (Phase 3) 4. The substrate file, versioned, with changelog 5. The oracle file and the coverage result 6. The drift report and the minimal-pair log 7. Re-test triggers: any model version change, any policy change, any new input field, any new production model If the user cannot run models now, deliver items 1 to 5 plus the exact drift-test commands and the folder layout for pasted runs, and state clearly that the substrate is **unvalidated** until Phase 6 passes. --- ## Worked example (condensed) Input from the user: You are an expert SDR manager with 20 years of experience. Review this lead and tell me if it is hot, warm, or cold. Be consistent. Do not hallucinate. Think step by step. [VP RevOps, 450-person B2B SaaS, last email reply 182 days ago, 3 pricing visits in 30 days, latest 12 days ago, webinar 64 days ago, no demo request] Phase 2 audit (excerpt): 0 of 11 substrate components present; 6 placebo lines; "hot/warm/cold," "good fit," "recently," and "engaged" unbound; open question at the end. Census finds the decisive gap: nothing says whether 182 days of email silence outweighs three pricing visits 12 days ago (class 7, aggregation, and class 8, precedence). Phase 3 policy (confirmed): RECENCY is set by the most recent engagement event in any channel; email opens are not engagement; "No" demo becomes "none on record." Phase 5: oracle grid of 414,720 boundary cases, PASS, every label reachable. Expected output, every run, every model: ICP_FIT=STRONG INTENT=HIGH RECENCY=ACTIVE TEMPERATURE=HOT ROUTE=AE_SAME_DAY Full files: `examples/lead_temperature_loose.md`, `examples/lead_temperature_substrate.md`, `examples/lead_temperature_oracle.py`. Full walkthrough with outputs: `references/worked-example.md`. ## Files | File | Use in | |---|---| | `scripts/audit_prompt.py` | Phase 2: lexical drift audit of any prompt | | `scripts/coverage_check.py` | Phase 5: proves the rules give one legal answer for every input | | `scripts/drift_test.py` | Phase 6: temporal, spatial, contextual drift report with SHA-256 hashes | | `references/drift-taxonomy.md` | Interpreting what kind of drift you have | | `references/ambiguity-census.md` | Phase 2: the 14 gap classes and placebo table | | `references/gtm-ontology-starters.md` | Phase 3: PROPOSED definitions for common GTM concepts | | `references/substrate-grammar.md` | Phase 4: writing rules with bad/good pairs | | `references/diagnosis-playbook.md` | Phase 7: divergence signature to patch | | `references/worked-example.md` | Full run of all phases on the lead example | | `assets/substrate-template.md` | Phase 4 skeleton | | `assets/oracle_template.py` | Phase 5 skeleton | | `examples/` | Lead temperature and acquisition: loose prompt, substrate, oracle |