Tokens spent running tasks in production are paid on every execution and never amortize. That pushes this kind of work onto an open inference stack, where the bill is set less by the GPU-hour price than by how well you tune it to your own workload.
Tokens spent building software are paid once and amortized across every user. Tokens spent running tasks in production are paid on every execution and never amortize. The usual case for running inference yourself compares the token price to the GPU-hour price, and that comparison keeps narrowing as prices fall. The stronger case is control - batching, caching, quantization, data placement, and what you can measure - which moves the bill far more than the hourly rate does, and is the only way to get latency and privacy on your own terms.
Runtime tokens are spent executing the task end to end, wherever that execution happens: inside a user request, or inside a system that runs without one - a trading loop, a fraud check, an automated decision in a pipeline. They may or may not involve generating code along the way. What binds them varies with the workload - latency in some, throughput in others - but the tokens are paid on every execution, and they are never amortized across anything.
Dev/research tokens are the other kind: the tokens spent building what ships - the application code, the harnesses, the agents themselves. They are spent once, offline, and amortized across every user and every execution. They are latency-tolerant: an agent that spends forty minutes and two million tokens getting a feature right is a good trade, because the result is reused indefinitely. Getting it right is the only constraint that matters.
Which kind they are depends on where you sit. The dev tokens you spend building software are runtime tokens to the lab serving them, paid on every request and bound by every constraint below.
One place the harnesses differ is in what checks the work. The dev-time harness optimizes for quality and measures success in whether the generated code passes tests. The runtime harness sits in the path of the work itself and has to be fast, predictable, and cheap, with verification that runs inside every execution.
Verification is where that difference shows up in the bill. At dev time a human or a test suite checks the output once, before it ships, and the cost of checking is amortized along with everything else. At runtime there is nobody to check it, so the check has to run inside the task: reading the output back against a schema, re-running a query to confirm a number, having a second and cheaper model grade the first. Each of those is more tokens, spent inside the same budget, on every execution. It is a large part of why a task costs tens of thousands of tokens rather than a few thousand, and it scales with how much you need to trust the result.
Dev tokens amortize. You spend them once and divide by every user who ever touches the feature, so the cost per user trends toward zero as the product grows. That curve is the reason software margins look the way they do. Runtime tokens do not amortize. They are spent per task, so the cost is flat at best per unit of work and in practice grows with use, because a more useful system gets used more. Runtime tokens put marginal cost back into software, which never had it.
That marginal cost also decides which tasks can exist at all. Many tasks that plan the request, call tools, and verify their own output at runtime are not viable at today’s prices and model capabilities, but will be at next year’s. Running a given level of capability gets roughly ten times cheaper every year. That is a16z’s estimate, measured at a fixed benchmark score, and a conservative one: Epoch AI’s measurements across six benchmarks put the median nearer fifty times a year, with a wide spread depending on the task. Each order of magnitude tends not to reduce the bill. It makes work viable that was not viable before.
Coding a workflow in advance was the affordable option, not the best way to describe what software should do. Every year open models get more capable and running them gets cheaper, another set of tasks crosses the line where executing them beats freezing them into code.
A token-metered API is a pure variable cost. You pay for what you use, nothing when idle, and the bill tracks volume exactly. Running inference yourself inverts that. Rented or owned, you are paying for capacity, and the bill arrives whether the hardware is busy or not. So the comparison is not really a price comparison. It is a utilization problem.
The arithmetic is worth doing once. A capable model on a token-metered API costs a few dollars per million tokens - call it $4 blended, since agentic traffic is mostly input. An eight-GPU H100 node rents for roughly $20 an hour, about $480 a day. Serving a 235B-parameter mixture-of-experts model at FP8, with roughly 22B parameters active per token, that node sustains on the order of 15,000 tokens per second across mixed prefill and decode, which is about 1.3 billion tokens a day if you keep it saturated.
Saturated, that is roughly $0.37 per million tokens, about a tenth of the API price. Nobody is saturated. At 30 percent utilization it is $1.20 per million. At 10 percent it is $3.70, which is API pricing plus the work of running your own hardware. So the crossover is a utilization threshold rather than a volume threshold, and it moves with how much hardware you commit: a single GPU serving a small model crosses over in the low millions of tokens a day, while a multi-node deployment needs hundreds of millions.
Which turns the cost question into a control question. Utilization is something you tune, not something you buy: batching policy, cache reuse, quantization, which model handles which step. The hourly rate is only the starting point.
The requirements change once tokens are in the critical path of production work. None of them are on a price list, but all of them affect the bill.
At dev time, the only thing you need from a model provider is a good model. At runtime, you need a controllable system. You need a latency floor and predictable tail latency. You need the data to stay inside your own boundary. At runtime the model sees whatever the task touches - customer records, internal systems, tool output - and if the model is somebody else’s, all of that leaves your infrastructure on every execution. In regulated environments that alone decides the question. You need control over batching policy, prefix and KV cache reuse across turns of the same interaction, speculative decoding, and quantization tradeoffs - because in a loop that runs thousands of times per user session, those are the difference between viable and not. And to optimize any of it for your traffic pattern specifically, you need to be able to profile it, down to the kernel.
Every one of those requirements asks for access the API does not give you, and an open inference stack is simply one that does. Open weights mean the model is yours to pin, quantize, distill, and keep serving after someone else would have deprecated it. An open inference engine means batching, scheduling, and cache policy are configuration rather than another company’s product decisions. An open orchestration layer means the workload runs wherever capacity is cheapest this quarter, or wherever the data is allowed to live, on whatever hardware. This is what that control actually requires, and a closed provider cannot offer it without giving up the abstraction its product is built on. It is also where the savings come from: the same levers that make the workload viable are the ones that move the cost per token.
There is also the question of what you are actually running. Identical model names do not guarantee identical behavior: OpenRouter, measuring billions of tool calls across providers serving the same weights at the same precision, found tool-calling accuracy and tool-use propensity varying more than benchmarks suggest, and converging only as providers tuned their inference stacks. They see no sign of anyone degrading quality deliberately. Nobody has to act in bad faith for an endpoint to behave differently than it did last month, and from the outside you cannot tell.
Prefix caching is a good example. A multi-step task re-sends the same system prompt, tool definitions, and accumulated state on every step, and the only reason that is affordable is that most of it never has to be recomputed. Providers do offer this, and it works - cached reads run around a tenth of the input price. But it comes with fixed terms: entries expire on a schedule you do not set, and holding them longer costs extra. Run the stack yourself and the cache is not billed at all. It is memory on a GPU you have already paid for, and you decide how long entries live and which worker holds them.
Speed is not the same as throughput. Batching raises tokens per second and raises time to first token along with it. A token-metered API picks that tradeoff for you. Running the stack yourself you choose it per workload, and you can choose differently for the interactive step and the background one.
Which settings are right depends on the traffic, not just the model. Prefix caching cuts the other way in some workloads: with high concurrency and short prompts that rarely repeat, the cache earns nothing, still costs something to maintain, and turning it off can multiply throughput. Batch size, scheduling policy, and speculative decoding all behave like that. Defaults are set for an average workload, and few real workloads are average. The only way to know which way each setting goes is to measure your own traffic where it actually runs, and that measurement has to reach parts of the stack an API does not expose.
This is a familiar pattern in infrastructure. Interactive, human-facing, low duty cycle work tolerates a closed product, because convenience is worth more than control when the machine is idle most of the time and the bill is small. Continuous, unattended, high volume work does not. The workload that runs continuously and defines your cost structure ends up on the open inference stack.
Dev tokens sit on the tolerant side of that line. Interactive, human-in-the-loop, quality over cost, spent in bursts by a small number of engineers. Closed frontier APIs are a fine answer there, probably the best one, and none of this changes that.
Runtime tokens sit on the other side. Unattended, high volume, latency- and cost-defined, running every hour of every day. That is the kind of workload that has always migrated to the open inference stack, and the same forces are pushing it there now.