The harness matters more than the model

Every few weeks a new model drops and the same question follows: can we finally let the agents run on their own? Wrong question. After running tens of different agents on different LLM models inside our own operations, the pattern is clear: the model determines how good a single decision is; the harness determines how much damage a bad one can do. Autonomy is not a property of the model. It is a property of the system around it.

Autonomy is a dial, not a switch. We run every agent workflow at one of three notches, and the notch is decided per workflow, not per model and not per team. The same model can sit at notch one in one workflow and notch three in another, because the failure modes and the blast radius are different.

Mode 1: AI-augmented

The human drives; the agent multiplies. An engineer works a ticket the normal way, but the agent does the exploration, drafts the diff, writes the tests, and chases down the docs. Every merge is owned by a human who read the change and can defend it. This is where every new workflow starts, and it is where anything ambiguous or high-stakes stays: architecture decisions, anything touching billing, anything customer-facing on day one.

The point of this mode is not just safety. It is calibration. You learn where the agent is reliably strong and where it confidently produces garbage, and you learn it while a human is still in the loop for every artifact.

Mode 2: Supervised agents

Here an agent takes a whole ticket end to end: picks it up from the tracker, plans, implements, tests, opens the pull request. But nothing lands unattended. A separate review agent — different prompt, sometimes a different model — audits the change against our standards and blocks on violations. Then a human validation gate sits before Done: someone checks the result in the real environment, not just the diff. Two automated opinions plus one human decision.

Most of our internal engineering throughput lives at this notch. It is the sweet spot: agents do the hours, humans spend minutes, and the review trail means we can reconstruct exactly why any change happened.

Mode 3: Autonomous operations

Scheduled loops that run without anyone watching: nightly maintenance, usage reporting, status sweeps, triage. No human in the loop — which is precisely why the engineering moves entirely into the guardrails. Every autonomous loop we run has hard budget caps, so a runaway agent hits a wall instead of a credit card. Every one has a kill-switch that a human can throw in seconds. Credentials are scoped to exactly the resources the loop needs, so the worst case is bounded by IAM policy rather than by good intentions. And everything writes an audit trail: what it read, what it decided, what it changed.

Trust is not a feeling here. Trust is an audit trail plus a kill-switch. If you cannot answer "what did it do last night and how do I stop it right now," the loop is not ready for this notch, regardless of how impressive the model demos.

Graduation is earned, one notch at a time

A workflow moves up the dial on evidence, never on vibes. Before we promote anything, we want a track record at the current notch: how many runs, how many escapes past review, what the failures looked like and whether the guardrails caught them. A workflow that has produced fifty clean supervised tickets is a candidate for a scheduled loop. A workflow that needed human correction last week is not, even if the newest model scores better on benchmarks.

The real engineering surface is not the prompt. It is:

• Permissions — what the agent can touch, enforced by scoped credentials, not by instructions
• Review gates — who or what must approve before an artifact becomes real
• Budgets — hard caps on spend, tokens, and runtime per loop
• Audit trail — a reconstructable record of every decision and action
• Kill-switch — a fast, tested way to stop everything

Models will keep getting better, and each upgrade shifts where a given workflow can sit on the dial. But the dial itself — the harness — is what lets you adopt those upgrades without betting the company on a benchmark. Build the harness first. Then turn the dial as fast as the evidence allows, and no faster.