Old frontier capability becomes a commodity quickly
Distillation, quantization, sparsity, better kernels, newer accelerators, and competition drive the same measured capability onto smaller and cheaper systems. New reasoning workloads can consume the savings.
Three numbers that locate the frontier
API price is not provider cost; MMLU is not every useful task; benchmark throughput is not interactive service quality.
Fix capability before claiming a cost curve
A valid curve holds quality, context, latency, and availability constant. Tokens per dollar without correctness can reward cheap failure; benchmark accuracy without serving constraints can reward unusable delay.
Smaller models can win by routing easy work locally and escalating hard cases.
Longer reasoning can trade additional compute for accuracy, turning inference demand into a variable chosen at runtime.
The headline metric sits on a system
Each layer can become the bottleneck even when the layer before it improves.
Model and data
Architecture, training distribution, post-training, and retrieval determine what quality is possible.
- Measure
- Task quality · robustness
- Failure mode
- Data and evaluation
Numerics and memory
Precision, sparsity, cache, and bandwidth determine how efficiently weights and activations move.
- Measure
- Bytes/token · joules/token
- Failure mode
- Memory bandwidth
Serving system
Batching, routing, speculative decoding, and scheduling trade throughput against latency.
- Measure
- Tokens/s · tail latency
- Failure mode
- Utilization
Application
Tools, verification, humans, and workflow design determine whether an answer creates value.
- Measure
- Cost per accepted outcome
- Failure mode
- Reliability
The irreducible unit is useful information, not a token
Physics sets distant switching and communication floors, but current system cost is dominated by moving data and maintaining utilization. Application value depends on correct, timely outcomes rather than generated symbols.
Cheap inference expands demand and exposes verification
As raw tokens become cheap, users ask for more context, reasoning, modalities, and agents. The bottleneck moves into trustworthy evaluation, tool execution, and human review.
Smaller specialists
Route tasks to the least expensive model that meets quality.
Better serving
Batch, cache, and schedule without violating latency.
Hardware-software codesign
Match numerics and memory hierarchy to workloads.
Outcome evaluation
Measure accepted work and downstream error cost.
An optimistic view, with conditions
Machine intelligence becomes metered infrastructure
Capability at yesterday's frontier will keep getting cheaper, while premium spend concentrates on difficult, verifiable, high-value tasks.
Sources, method, and boundaries
Price and benchmark results are measured but use different boundaries. No API list price is treated as underlying production cost.
