As models get capable enough to run whole tasks in production, the tokens that do it are paid on every execution, bounded by a latency or throughput budget, and never amortized. Those constraints decide where and how inference has to run.
Models are getting capable enough to run a whole task end to end: plan the request, call tools, check the result, answer. As they get faster and more capable, more work runs that way. Every one of those tokens is paid on every execution, whether a user is waiting on it or it runs unattended in the background. The economics of it are not only a matter of price. Latency, control over the runtime, and what you can ship at all are part of the same calculation, and together they are what make an open inference stack the sensible way to run it.
Dev tokens are the tokens spent building what ships - the application code, the harnesses, the agents themselves. They are spent once, offline, and amortized across every user and every execution. They are latency-tolerant: an agent that spends forty minutes and two million tokens getting a feature right is a good trade, because the result is reused indefinitely. Getting it right is the only constraint that matters.
Runtime tokens are spent executing the task end to end, wherever that execution happens: inside a user request, or inside a system that runs without one - a trading loop, a fraud check, an automated decision in a pipeline. They may or may not involve generating code along the way. Getting it right matters just as much here, but it has to happen inside a budget, on every execution. What binds it varies with the workload - latency in some, throughput in others - but it is paid on every execution, and it never amortizes.
The harnesses differ mainly in what checks the work. The dev-time harness optimizes for quality and measures success in whether the generated code passes tests. The runtime harness sits in the path of the work itself and has to be fast, predictable, and cheap, with verification that runs inside every execution.
Verification is where that difference shows up in the bill. At dev time a human or a test suite checks the output once, before it ships, and the cost of checking is amortized along with everything else. At runtime there is nobody to check it, so the check has to run inside the task: reading the output back against a schema, re-running a query to confirm a number, having a second and cheaper model grade the first. Each of those is more tokens, spent inside the same budget, on every execution. It is a large part of why a task costs tens of thousands of tokens rather than a few thousand, and it scales with how much you need to trust the result.
Dev tokens amortize. You spend them once and divide by every user who ever touches the feature, so the cost per user trends toward zero as the product grows. That curve is the reason software margins look the way they do. Runtime tokens do not amortize. They are spent per task, so the cost is flat at best per unit of work and in practice grows with use, because a more useful system gets used more. Runtime tokens put marginal cost back into software, which never had it.
That marginal cost also decides which tasks can exist at all. Many tasks that plan the request, call tools, and verify their own output at runtime are not viable at today’s prices and model capabilities, but will be at next year’s. Running a given level of capability gets roughly ten times cheaper every year. Each order of magnitude tends not to show up as a smaller bill. It shows up as work that was not worth running becoming worth running.
Coding a workflow in advance won because it was affordable, not because it was the best way to describe what the software should do. Every year open models get more capable and running them gets cheaper, another set of tasks crosses the line where executing them beats freezing them into code.
A token-metered API is a pure variable cost. You pay for what you use, nothing when idle, and the bill tracks volume exactly. Running inference yourself inverts that. Rented or owned, you are paying for capacity, and the bill arrives whether the hardware is busy or not. So the comparison is not really a price comparison. It is a utilization problem.
The arithmetic is worth doing once. A capable model on a token-metered API costs a few dollars per million tokens - call it $4 blended, since agentic traffic is mostly input. An eight-GPU H100 node rents for roughly $20 an hour, about $480 a day. Serving a 235B-parameter mixture-of-experts model at FP8, with roughly 22B parameters active per token, that node sustains on the order of 15,000 tokens per second across mixed prefill and decode, which is about 1.3 billion tokens a day if you keep it saturated.
Saturated, that is roughly $0.37 per million tokens, about a tenth of the API price. Nobody is saturated. At 30 percent utilization it is $1.20 per million. At 10 percent it is $3.70, which is API pricing plus the work of running your own hardware. So the crossover is a utilization threshold rather than a volume threshold, and it moves with how much hardware you commit: a single GPU serving a small model crosses over in the low millions of tokens a day, while a multi-node deployment needs hundreds of millions.
Cost is not the whole calculation. Capability decides whether the comparison happens at all - if a step genuinely needs a frontier model, there is nothing to compare and you pay the API price. The question is how many steps in a task actually need one, and for most of them the answer is none.
Speed is not the same as throughput. Batching raises tokens per second and raises time to first token along with it. A token-metered API picks that tradeoff for you, tuned to a tenant mix that is not yours. Running the stack yourself you choose it per workload, and you can choose differently for the interactive step and the background one.
The requirements change once tokens are in the critical path of production work, and most of what changes has nothing to do with price.
At dev time, the only thing you need from a model provider is a good model. At runtime, you need a controllable system. You need a latency floor and predictable tail latency. You need the data to stay inside your own boundary. At runtime the model sees whatever the task touches - customer records, internal systems, tool output - and if the model is somebody else’s, all of that leaves your infrastructure on every execution. In regulated environments that alone decides the question. You need control over batching policy, prefix and KV cache reuse across turns of the same interaction, speculative decoding, and quantization tradeoffs - because in a loop that runs thousands of times per user session, those are the difference between viable and not. And to optimize any of it for your traffic pattern specifically, you need to be able to profile it, down to the kernel.
Every one of those requirements asks for access the API does not give you, and an open stack is simply one that does. Open weights mean the model is yours to pin, quantize, distill, and keep serving after someone else would have deprecated it. An open inference engine means batching, scheduling, and cache policy are configuration rather than another company’s product decisions. An open orchestration layer means the workload runs wherever capacity is cheapest this quarter, or wherever the data is allowed to live, on whatever hardware. This is what that control actually requires, and a closed provider cannot offer it without giving up the abstraction its product is built on.
Prefix caching is the clearest case. A multi-step task re-sends the same system prompt, tool definitions, and accumulated state on every step, and the only reason that is affordable is that most of it never has to be recomputed. Providers do offer this, and it works - cached reads run around a tenth of the input price. But it arrives as a product with terms attached: entries expire on a fixed schedule, and holding them longer costs extra. Run the stack yourself and the cache stops being a line on the invoice. It is memory on a GPU you have already paid for, held as long as you decide to hold it, on a worker you can pin the task to.
Which settings are right depends on the traffic, not just the model. Prefix caching cuts the other way in some workloads: with high concurrency and short prompts that rarely repeat, the cache earns nothing, still costs something to maintain, and turning it off can multiply throughput. Batch size, scheduling policy, and speculative decoding all behave like that. Every default is a bet on what an average workload looks like, and no real workload is average. The only way to know which way each setting goes is to measure your own traffic where it actually runs, and that measurement has to reach parts of the stack an API does not expose.
This is a familiar pattern in infrastructure. Interactive, human-facing, low duty cycle work tolerates a closed product, because convenience is worth more than control when the machine is idle most of the time and the bill is small. Continuous, unattended, high volume work does not. The workload that runs continuously and defines your cost structure ends up on the open stack.
Dev tokens sit on the tolerant side of that line. Interactive, human-in-the-loop, quality over cost, spent in bursts by a small number of engineers. Closed frontier APIs are a fine answer there, probably the best one, and none of this changes that.
Runtime tokens sit on the other side. Unattended, high volume, latency- and cost-defined, running every hour of every day. That is the profile that has always migrated to the open stack, and the same forces are pushing it there now.
The frontier model is not what many tasks need anyway. They need a model that is good enough at the step in front of it, running under a system you control, at a cost that survives being multiplied by every interaction. That is the threshold open models have crossed. The usage data already shows the split: OpenRouter’s study finds closed models capturing the high-value tasks while open models take the high-volume, lower-value ones.