Local Inference as Token-Budget & Outage Insurance (L4)
Run a small local model (Gemma, Mistral, Llama via Ollama) as redundancy for routine work, privacy-sensitive analysis, and GPU brownout days. Stay productive when Claude/OpenAI go down. From John Williams (FXOps) on the GTM AI Podcast.
The steps
- 01
Pick a small model that fits your machine
Tool: Ollama
Gemma 2 9B, Mistral 7B, or Llama 3.1 8B all run on a modern MacBook. Modern small models are no longer the barrier they were a year ago.
- 02
Route low-stakes work locally by default
Tool: Manual / router
Batch summarization, drafts, name-extraction, tagging, all local. Reserve frontier-model calls (Opus, GPT-5) for reasoning-heavy tasks. The cost delta compounds fast across an agent fleet.
- 03
Use local as your outage fallback
Tool: Ollama
When Claude is down (it happens), keep shipping. The browser extension above lets you take the same thread to your local model without losing context.
- 04
Run anything privacy-sensitive offline
Tool: Ollama (no network)
Regulated work, board prep, M&A research, compensation analysis, disconnect from the internet, run the inference, you're the only audit trail. "Token authority" = you can do the work no matter what the cloud is doing.
Tools in this playbook
- Ollama
- Manual / router
- Ollama (no network)
Next playbooks
Unfamiliar terms are defined in the AI and Revenue Dictionary. Related frameworks live in the framework library.