Inference tiering is not an optimization, it’s a design decision

Model tiering is always presented as a cost table: local is free and dumb, cloud is expensive and smart, route accordingly. The table is true and misses the point. Tiering is a software-architecture primitive. The local model’s job is not to answer questions; it is to decide who answers. The intelligence of the system lives in the routing layer.

Parent discussions

• Isidore — Agentic Operating System — the thought experiment that raised the question

Connected concepts

• AI is an interface shift, and the window is the escalation path — routing which model answers and routing which modality responds are the same decision made twice
• AI Pair Programming Workflow — tier selection is a parse-phase decision the system makes about itself

The point

Standard tiering is a three-tier cost table: a local model that is free and dumb, a cheap cloud model for medium work, a premium cloud model for hard reasoning. Read as a cost table, this is three price points. But look at the local tier’s actual use: routing decisions, classification, status checks. The local model is mediocre at hard questions, and that is fine, because answering hard questions is not its job. It has to be good at one thing: knowing what kind of question it is holding. Everything follows from whether it gets that right.

Nobody calls a kernel scheduler a cost optimization. The scheduler decides where and when your program runs, and that decision is the operating system in a real sense. The routing classifier occupies the same position. The tiers behind it are interchangeable: swap the premium tier for whatever is best next quarter and the system is unchanged. Swap the router’s judgment and you have a different system. The failure modes show it too. A misroute upward wastes money, but a misroute downward delivers a wrong answer confidently and cheaply, which is the most expensive thing a system can do. The quality of the tiers bounds how good an answer can be; the quality of the router determines whether you ever see it.

A second-order effect the cost table hides: when eighty percent of daily tasks run at zero marginal cost and zero perceptible latency, you stop rationing the interface. You ask the small questions, the status checks, the file operations, the “remind me what this flag does”, that you would never spend a cloud call on. Free-and-instant changes what the system is for. That behavioral shift is designed in, not discovered. Which is the whole claim of this note. The router is the kernel; the rest is drivers.

Open questions

• When the local model misclassifies, how does the system recover gracefully? Retry upward on low confidence, or learn from the correction?
• Does the router deserve its own fine-tune on personal traffic, given that its distribution is one household’s questions, not the internet’s?
• Where does routing state live? A router that remembers past routings is starting to look like an agent itself. Is that a layer violation or the design?