Name the role, not the model
Not so long ago, switching a model in our agent fleet meant a grep, a pull request and a hope that nobody had pasted the old model name into a script we’d forgotten about. Someone always had. So we made a rule: no skill, script, prompt or worker names a model. They name a role, such as “the developer”, “the second reviewer” or “the cheap one for polling”, and a registry decides which model fills it today.
The rule is enforced, not hoped for. A guard test fails the build if a new model literal shows up in code. This post covers what sits behind the role name, how a request resolves, and the incidents that taught us most of the details.
What a role is
We have about 35 roles: dev, frontend dev, primary and second code review, mechanical, orchestration, digest, validation and so on, plus a couple of special kinds for video and vision work. Each role is a model, an effort level, an ordered fallback list and a kind. That’s all. “Dev” is not a model, it’s a job description with a current holder and a bench.
The payoff is boring in the best way. When a better or cheaper model appears, a swap is one registry write. Nobody edits a skill, nobody redeploys a worker, and every caller picks up the change on its next resolve.
How a request resolves
The registry lives behind a small admin API on top of DynamoDB. A job asks for a role. The registry resolves it through three scopes, global first, then project, then worker, with the narrower scope winning. Projects can also carry deny lists, and that is where it gets interesting for us.
One client project denies every non-Anthropic provider. In that project “dev” quietly resolves to a different model than everywhere else, and no skill had to know. Deny lists apply to every rung of the fallback ladder, so a fallback can’t sneak a forbidden vendor back in. A per-job override that names a denied model is ignored entirely, not downgraded. Compliance and customer constraints are data in the registry rather than a convention people have to remember.
The command-line client resolves online first and refreshes a local cache as it goes. If the registry is unreachable, an embedded copy of the resolver reads that cache instead. That copy is parity-tested against the real one, and it warns once the cache is older than 24 hours. One rule has no exception: if the registry rejects our credentials, we never answer from cache. A revoked caller shouldn’t keep working because a file happened to be on disk.
The dispatcher in front
Scripts don’t usually ask for roles directly. They ask for a job, and a loopback-only model dispatcher maps about 35 job rules to roles, sometimes conditionally on state, urgency or project. It also supports per-job overrides that expire on their own, so “run this job on that model for a day” can’t turn into a permanent surprise. Fallback ladders can compose across roles.
The answer carries the backend, model, effort, fallbacks, a TTL and a decision id, so any behaviour can be traced back to a decision. The dispatcher re-verifies against the registry every 15 minutes and alerts after six hours of staleness. If someone breaks the rules file, it keeps the last good version and alerts rather than failing open or closed.
Rules we encoded because we got them wrong
- Code review roles sit on different providers by design, and a review leg never runs on the model that built the change. That leg takes its next fallback instead.
- Human-facing roles reject the smallest model class, both when the role is written and again when it resolves. Changing one requires an explicit authorization quote.
- Retired models are dropped from stored ladders at resolve time. If nothing allowed remains, the API answers “no allowed model” with an HTTP 409. It does not guess.
The gateway
For anything that isn’t a direct call, we run a self-hosted LiteLLM gateway on one small ARM VM, about $17 a month. The client contract is two environment variables: a base URL and a per-lane virtual key. Claude Code speaks its normal API and has no idea anything changed. Rollback is unsetting both variables.
Per-lane keys carry budgets, which gives us attribution for free: we can see which lane spent what. A usage tracker polls spend logs every 60 seconds, and it deliberately reads 60 minutes behind its cursor, because rows are written when a call completes but stamped when it started. Role aliases at the gateway let us repoint a lane by config alone.
One thing we chose not to do: we don’t pool several subscription accounts behind one gateway. That is the limit-evasion pattern providers ban, and we’d rather hit a limit than lose an account.
Four ways it bit us
First, an emergency ladder put the most expensive model first on a six-hour pin after a refusal. Unattended, it burned over $35 a day. Ladders are now cheap-first, and when a pin’s TTL expires we re-probe the primary instead of riding the pin out.
Second, one provider signals quota exhaustion with HTTP 403 rather than 429. Our classifier only knew 429, so the lane looked healthy for an entire billing cycle while quietly failing over. The classifier now knows about that provider’s dialect.
Third, a shared key hit its aggregate budget cap. The fallback failed with the identical error, because the wall was the key’s budget and not a provider quota. Pre-flight checks now cover key budgets as well as provider limits.
Fourth, one provider rejects document content blocks. We pass that error through. Silently stripping the block would make the request succeed on a different question than the one asked, and hidden request mutation is worse than a visible failure.
The trade-off
Roles by reference make a model swap a one-line change, but they add a runtime dependency on the registry. We mitigate that with the cache, the embedded resolver and staleness alerts, and we’d still rather have the dependency than a hundred copies of a model name. The other cost is comprehension. Layered scopes plus deny lists plus composed ladders are powerful and hard to hold in your head. So resolution has to be inspectable: resolve --json prints the ladder, where each value came from, and which rungs were removed and why.
What we’d tell you to copy
- Put model names in exactly one place and add a test that fails when they appear anywhere else.
- Encode customer and compliance constraints as data that every rung of every ladder obeys, not as instructions people have to remember.
- Fail loudly: answer “no allowed model” instead of guessing, and pass provider errors through instead of rewriting requests.
- Make every fallback cheap-first, and make every degraded path alert someone, including the ones that look healthy.
None of this is clever. It’s the same idea as naming an interface instead of a concrete class, applied to a part of the stack that changes monthly. The models will keep changing. What we want is for that to be a registry write and a quiet afternoon, not a migration.