Support deflection with AI search (L3)

The steps

1. 01

   Audit and prune the Knowledge Base

   The foundation of any AI support tool is the quality of your existing Knowledge Base (KB). Before turning on any automation, the Support Manager must audit articles for clarity and accuracy, as AI like Intercom Fin or Zendesk AI will "hallucinate" or confidently provide wrong information if your articles are outdated. • Start by exporting your KB article list to a CSV. Sort by "Last Updated" and prioritize refreshing any article older than 6 months. • Ensure articles use structured headings (H1, H2) and clear, declarative sentences. AI parses bullet points better than dense paragraphs. • Remove conflicting information. If one article says "Refunds take 5 days" and another says "7 days," the AI will struggle. • Use an "Internal Notes" tag for any private info you do not want the AI to index. • Time Estimate: 10-15 hours depending on KB size. • Common Pitfall: Thinking the AI can "read between the lines." It can't. If the answer isn't explicitly written, the AI can't invent it safely. • Definition of Done: A spreadsheet of the top 50 most-viewed articles confirmed as 100% accurate and formatted for readability.

2. 02

   Limit AI scope to a single pilot area

   Attempting a full-scale AI rollout often leads to high friction. Instead, the RevOps or CS Lead should select a single, low-risk product area or "Collection" to serve as a pilot. • In Intercom, go to Fin AI > Content and select "Sync specifically selected folders" rather than your entire Help Center. • Choose a category like "Billing and Subscriptions" or "Basic Troubleshooting" where answers are objective and stable. • Set the "Audience" settings to only show the AI bot to a small percentage of your traffic (e.g., 10%) or to a specific segment like "Free Trial Users" to limit risk. • Owner: RevOps Lead. Time: 1 hour for setup. • Common Pitfall: Launching on technical API documentation first. AI struggles with complex code syntax compared to simple policy questions. • Definition of Done: The AI agent is live and Restricted only to the selected product folders and a specific user segment.

3. 03

   Configure confidence thresholds and hand-offs

   You must define when the AI should step back and involve a human. This prevents "looping" where a frustrated customer cannot get past the bot. • In your AI agent settings (e.g., Zendesk Answer Bot or Intercom Fin), locate the "Confidence Threshold" or "Hand-off" tab. • Set the threshold to 0.6 or "Balanced." This means if the AI is less than 60% sure it has the right answer, it will immediately offer to route the user to a human agent. • Configure a "Human Hand-off" workflow. In Intercom, this is an "Action" block that triggers "Assign to Team: Support." • Ensure the AI always provides a "Talk to a person" button at the bottom of every response. • Owner: CS Manager / Tool Admin. Time: 2 hours. • Common Pitfall: Setting the confidence too high (causing it to never answer) or too low (causing it to give "garbage" answers). 0.6 is the industry-standard sweet spot for starting. • Definition of Done: A test conversation where asking an unanswerable question triggers an immediate, seamless routing to the live support inbox.

4. 04

   Set up attribution and deflection tracking

   To measure success, you need to track how often the AI actually solves the problem without human intervention. • Go to your platform's Analytics dashboard (Intercom Reports > Fin or Zendesk Explore). • Create a custom report focusing on "Deflection Rate" (Users who closed the chat after an AI response without asking for a human) and "Resolution Rate." • Set up a "Negative Feedback" alert. Most tools allow you to tag conversations where a user clicks the "Thumbs Down" icon. • Review the "AI CSAT" specifically. AI-handled tickets should maintain a CSAT within 10% of your human agents. • Owner: RevOps Analyst. Time: 3 hours for dashboard setup. • Common Pitfall: Forgetting that a "Closed" chat doesn't always mean a "Solved" chat. Users sometimes give up in frustration. Always cross-reference deflection with the "Negative Feedback" rate. • Definition of Done: A live dashboard showing Deflection Rate, CSAT, and Volume Handed to Human.

5. 05

   Run weekly "Bad Answer" triage sessions

   AI optimization is an iterative process. You must hold a weekly "Conversation Review" to identify where the bot is failing. • Filter your inbox for "Handed over from AI" and "Rated 1-star." • Look for "Clustering" trends. For example, if 20 people asked about "Refunds" and the AI failed, check if the KB article for refunds is missing a keyword the users are using (e.g., "Cash back"). • "Patch" the KB immediately based on these insights. If the AI is hallucinating, add a "Custom Answer" or "Hard Guardrail" in the settings to explicitly tell the bot: "If asked about X, always say Y." • Owner: Support Lead / Content Strategist. Time: 2 hours weekly. • Common Pitfall: Ignoring the "Missing Content" report provided by the AI. This report tells you exactly what customers are searching for that you haven't written about yet. • Definition of Done: A weekly log of 5-10 KB articles updated specifically based on AI failure points.

Next playbooks

• CSM call summaries via Gainsight + AI (L2)
• AI-generated QBR decks (L3)
• Intercom Fin AI for support deflection (L3)
• Customer health scoring with AI signals (L4)