// teardown
The AI harness
The machine around the model.
// where it sits
The harness is one layer of a blueprint I've shipped three times — for a fintech, a real-estate marketplace, and an enterprise GTM platform. Data infrastructure first: capture every event, unify it into one source of truth, operate it from a command center. Then the intelligence layer on top, drawing context from that core and feeding every action back into it.
// the whole system
data ⇄ intelligence
Transactional core
Intelligence layer
every action becomes new signal
// shipped in production
bartr
172 tables · 774 backend fns · daily AI valuations
Ryse · Atlas
intelligence layer on the transactional core · agentic back office
LightBox GTM
six systems unified · natural-language command center
zoom in · The AI harness
// how a request flows
one model · one queue · one audit trail
Request
drawer · Slack
Router
picks a mode
agent loop · reason ⇄ act
tools run as the user · RBAC re-checked
Stream
results back
Live
in the UI
// one router, purpose-built modes
gated per task
Copilot
in-product & Slack
- tools
- read · gated writes
- access
- acts as the user
- context
- CRM + warehouse
Author
PRDs, specs & issues
- tools
- docs · issue tracker
- access
- write to Linear
- context
- product + roadmap
Coder
autonomous coding
- tools
- repo · CI · browser
- access
- short-lived token
- sandbox
- isolated · egress-capped
Analyst
NL → warehouse
- tools
- read-only SQL
- access
- read-only
- context
- the warehouse
Each mode gates its own tools, permissions, context window, and system prompt — with compute and memory scaled to the task. Same model, purpose-built per job.
// overview
Anyone can call a model. The hard part is the runtime around it — a router that picks a purpose-built mode, tools that run with the caller's own identity, writes that arrive as proposals rather than actions, and an audit trail that makes every run replayable.
// how it works
01 · route
One router, not one prompt
Every request — typed in the product, sent in Slack, or fired by a schedule — hits a single router first. It classifies the task and dispatches to a mode. That indirection is what keeps a growing tool surface from collapsing into one bloated system prompt that gets slower, more expensive, and less accurate with every capability added.
02 · scope
A mode is a permission boundary
A mode bundles four things: its tools, its permissions, its context window, and its system prompt — with compute and memory sized to the task. Analyst gets read-only SQL and nothing else. Coder gets a repo, CI, and a short-lived token in an isolated sandbox. The boundary is enforced at dispatch, not suggested in a prompt.
03 · retrieve
Retrieval is a strategy, not a vector search
Most "RAG" answers are bad because everything gets embedded and nothing gets ranked. Structured facts come from the warehouse by query, not similarity — a count should be a count. Prose is chunked on semantic boundaries with its source and recency carried alongside, re-ranked before it enters the window, and cited in the answer. What can be resolved deterministically never reaches the model as a guess.
04 · reason
The loop runs as the user
Inside a mode the agent reasons in a loop — call a tool, read the result, decide again. Every tool call carries the caller's identity and scopes, re-checked on each invocation rather than trusted from the start of the turn. Reads happen directly. Writes are drafted as proposals that a human approves, so the blast radius of a wrong answer is a rejected suggestion.
05 · return
Streamed into the product, not into a chat window
Output lands where the work already is: a drafted reply in the composer, a populated table, a Slack thread, a filled-in issue. The chat surface is one client of the harness, not the product. That distinction is what makes the intelligence feel embedded rather than bolted on.
06 · evaluate
Evals are the regression suite
Prompts and tool definitions are code, and they regress like code. Real traced runs become fixtures with graded expectations — did it pick the right mode, call the right tools in the right order, cite real sources, refuse what it should refuse. A model or prompt change runs the set before it ships. Without this, every improvement is a guess and every regression is discovered by a user.
07 · fail
Failure modes are designed, not discovered
The interesting engineering is in what happens when it goes wrong. Tool calls are idempotent and retried with backoff. Loops have step and token ceilings that end in a partial answer plus what it was still missing, never a hang. Low-confidence retrieval says so instead of confabulating. Every run — success or failure — lands on the same queue and audit trail, which is what makes an incident reconstructable a week later.
// decisions
The parts that were a call, not a default — and what the alternative would have cost.
A harness with a router and scoped modes
One agent with every tool attached
A single agent degrades as tools accumulate: latency and cost climb with the tool surface, and accuracy falls as the model picks between near-duplicate options. Modes keep each decision small, and adding a capability means adding a mode rather than growing a prompt everyone is afraid to edit.
Writes as human-approved proposals
Autonomous writes with a rollback path
Rollback assumes you notice. In a back office the expensive failures are quiet and plausible — a wrong field on the right record. Gating writes behind an approval keeps a human on the only step that is hard to undo, and turns the agent's speed advantage into review throughput rather than risk.
Tools run with the caller's identity
A privileged service account
A service account makes the agent the union of everyone's permissions, and makes the audit log say the robot did it. Passing through the caller's identity means existing RBAC keeps working unchanged, an agent can never read what its user could not, and every action attributes to a person.
Disposable sandboxes with capped egress
Running autonomous work on shared infrastructure
Autonomous coding executes code the model wrote against dependencies it chose. Treating that as untrusted by default — fresh sandbox, short-lived credentials, an allowlist on the way out — is cheaper than trying to decide, per run, whether this one is safe.
// guarantees
Runs as the user
Tools act with the caller's identity and scopes, re-checked on every call. Reads by default; writes are proposals, gated behind a human approval.
Isolated & auditable
Autonomous work runs in disposable sandboxes with capped egress — and every tool call lands on one shared queue and audit trail.
Proactive, not reactive
Agents watch the warehouse for signals — a dip, a stall, a spike — investigate, and surface what they found before anyone asks.