Fix capability before claiming a cost curve
A valid curve holds quality, context, latency, and availability constant. Benchmark accuracy without serving constraints can reward unusable delay just as easily as tokens per dollar without correctness can reward cheap failure.
What does an accepted answer actually cost?
The model fixes a task, then separates raw token cost from the verification and retry overhead needed to trust the answer, echoing the article's own frame: compute plus memory plus serving, divided by accepted task outcomes.
$0.0002 in raw tokens × 1.1× verification overhead
- Raw token cost
- $0.0002
- Verification & retry overhead
- 1.1×
- Cost per 1,000 accepted outcomes
- $0.19
September 2026 API pricing spans roughly $0.02–0.20 per million input tokens for budget and mainstream models up to $5–10 for flagship models, with output typically priced 5–6× higher than input. Move the verification slider and notice how a 25× sticker-price gap between budget and flagship models can widen to far more once retries and review are counted, the mechanism behind the article's claim that cheap tokens push scarcity into trustworthy evaluation.
Calculation and boundaries
Raw token cost = (prompt tokens ÷ 1,000,000 × input price) + (output tokens ÷ 1,000,000 × output price). Cost per accepted outcome = raw token cost × verification/retry overhead multiplier. Excludes serving infrastructure cost, batching discounts, caching, and the underlying provider cost of compute: API list price is not production cost. Verification overhead is an illustrative multiplier representing retries, tool calls, and human review, not a benchmarked figure for any specific task.
Four layers between a model and a trusted answer
Cheap arithmetic does not automatically become a cheap accepted answer: each layer above the model can add its own cost.
Model and data
Architecture, training distribution, post-training, and retrieval determine what quality is possible.
- Measure
- Task quality · robustness
- Failure boundary
- Data and evaluation gaps cap achievable quality regardless of how cheaply the model runs.
Where the frontier moves
Post-training and retrieval methods that keep raising quality at fixed model size.
Numerics and memory
Precision, sparsity, cache, and bandwidth determine how efficiently weights and activations move.
- Measure
- Bytes/token · joules/token
- Failure boundary
- Memory bandwidth bounds throughput regardless of how much compute is available.
Where the frontier moves
Lower precision and sparsity that keep shrinking bytes moved per generated token.
Serving system
Batching, routing, speculative decoding, and scheduling trade throughput against latency.
- Measure
- Tokens/s · tail latency
- Failure boundary
- Utilization determines whether cheap compute translates into cheap service.
Where the frontier moves
Serving techniques that raise utilization without violating latency requirements.
Application
Tools, verification, humans, and workflow design determine whether an answer creates value.
- Measure
- Cost per accepted outcome
- Failure boundary
- Reliability failures mean cheap tokens can still produce an expensive mistake.
Where the frontier moves
Verification and tool-use design that catches errors without re-running the whole task.
The irreducible unit is useful information, not a token
Physics sets distant switching and communication floors, but current system cost is dominated by moving data and maintaining utilization. Application value depends on correct, timely outcomes rather than generated symbols.
Cheap inference expands demand and exposes verification
As raw tokens become cheap, users ask for more context, reasoning, modalities, and agents. The bottleneck moves into trustworthy evaluation, tool execution, and human review.
Smaller specialists
Route tasks to the least expensive model that reliably meets the required quality bar.
Better serving
Batch, cache, and schedule requests without violating the latency the application actually needs.
Hardware-software codesign
Match numerics and memory hierarchy to the specific workload rather than a generic configuration.
Outcome evaluation
Measure accepted work and downstream error cost, not tokens generated or benchmark score alone.
Who is building what
Inference ASICs, speculative decoding, multi-head latent attention, low-bit quantization (FP4/FP8), and hardware compilers drive down cost per token. Search the record, or filter by efficiency layer.
NVIDIATensorRT-LLM & Blackwell NVL72Second-generation Transformer Engines supporting native microscopic FP4 and FP8 quantization, paired with TensorRT-LLM in-flight batching and KV-cache compression
- Reported evidence
- Dominant market platform powering hyperscale AI inference; demonstrated up to 30× throughput speedup on Blackwell versus earlier architectures.
- Announced next step
- Driving inference cost per token down by orders of magnitude while supporting 100k+ token real-time context generation.
- Unresolved risk
- High accelerator capital acquisition costs, 120 kW rack thermal power envelopes, and software ecosystem vendor lock-in via CUDA.
GroqLanguage Processing Unit (LPU)Deterministic, single-core Tensor Streaming Processor (TSP) architecture with massive on-chip SRAM (230 MB per chip) and zero off-chip DRAM latency
- Reported evidence
- Commercial cloud API delivering industry-record generation speeds exceeding 500–800 tokens per second on open models (Llama-3).
- Announced next step
- Ultra-low-latency real-time voice, code generation, and multi-turn agentic loops requiring immediate token streaming.
- Unresolved risk
- Extremely high silicon area per gigabyte of memory (SRAM density limits) requiring hundreds of interconnected chips to hold a single large model.
Cerebras SystemsCS-3 Wafer-Scale EngineWafer-scale single-piece silicon processor with 900,000 AI cores and 44 GB of on-wafer SRAM delivering 21 petabytes/second memory bandwidth
- Reported evidence
- Commercial inference service setting world records for token generation velocity on open frontier models, outperforming multi-GPU clusters.
- Announced next step
- Replacing sprawling GPU clusters with single-wafer compute nodes for extreme-speed inference serving.
- Unresolved risk
- Custom packaging, power delivery, and water-cooling infrastructure required for a single 23 kW wafer-scale processor package.
DeepSeekDeepSeek-V3 & R1 (MLA)Multi-head Latent Attention (MLA) compressing the KV-cache by an order of magnitude, paired with fine-grained Mixture-of-Experts (MoE) activating only ~37B out of 671B parameters
- Reported evidence
- Trained frontier-grade reasoning models for a fraction of traditional compute costs; released models delivering commercial API pricing well below Western frontier APIs.
- Announced next step
- Democratizing high-capability reasoning and coding models through algorithmic and architectural efficiency rather than brute-force scaling.
- Unresolved risk
- Quantization degradation on complex algorithmic reasoning tasks, and long-term hardware optimization on non-NVIDIA silicon.
Together AI / vLLM ProjectPagedAttention & FlashAttentionVirtual memory management for KV-caches (PagedAttention) eliminating memory fragmentation, combined with speculative decoding and kernel fusion
- Reported evidence
- vLLM is the ubiquitous open-source standard powering commercial cloud inference, increasing serving concurrency and throughput by 2–4× on identical hardware.
- Announced next step
- Universal low-latency inference serving software running seamlessly across heterogeneous silicon architectures.
- Unresolved risk
- Managing memory fragmentation and scheduling overhead under bursty, highly variable prompt length distributions.
Mistral AISmall & MoE ModelsHigh-efficiency open-weights models and dense/MoE architectures (Mixtral 8x7B/8x22B) optimized for high parameter efficiency and distillation
- Reported evidence
- Models match or exceed earlier generation flagship model capabilities on common benchmarks at a small fraction of active parameter count and inference cost.
- Announced next step
- Edge-deployable and on-premise high-throughput models running efficiently on commodity enterprise servers.
- Unresolved risk
- Performance ceilings on highly complex multi-step mathematical and software architecture planning versus trillion-parameter dense models.
ModularMAX Engine & Mojo CompilerNext-generation unified AI compiler and runtime (MAX) and systems language (Mojo) optimizing kernel execution and memory management across heterogeneous CPUs and GPUs
- Reported evidence
- Commercial benchmarks demonstrate significant throughput speedups on PyTorch and TensorFlow models without manual CUDA C++ kernel rewrites.
- Announced next step
- Decoupling AI application software completely from underlying semiconductor hardware architectures.
- Unresolved risk
- Competing against heavily entrenched proprietary optimization libraries (NVIDIA TensorRT, AMD ROCm) backed by massive ecosystem inertia.
MLCommonsMLPerf Inference BenchmarksOpen industry consortium standardizing rigorous benchmarking of AI model inference across latency, throughput, energy efficiency, and cost
- Reported evidence
- Bi-annual published results independently measuring performance across server, edge, and mobile accelerators from all major semiconductor vendors.
- Announced next step
- Providing unvarnished, reproducible price-performance transparency across cloud and on-premise AI acceleration hardware.
- Unresolved risk
- Benchmark gaming through non-representative optimizations (extreme pruning, unrealistic batch sizes) that do not reflect production deployment.
Advertised tokens-per-dollar metrics often assume batch processing on small quantized models; interactive low-latency serving with long context windows incurs heavy KV-cache memory overhead.
The optimistic view, with conditions
Machine intelligence becomes metered infrastructure
Capability at yesterday's frontier will keep getting cheaper, while premium spend concentrates on difficult, verifiable, high-value tasks.
Yesterday's frontier keeps falling
Distillation and better serving push old capability onto smaller, cheaper systems every year.
Spend concentrates on verification
The hardest, highest-value tasks pay for reliability and evidence, not just raw tokens.
Measure the accepted outcome
Cost per accepted outcome, not cost per token, is what determines whether the curve is real progress.
What a real cost-of-intelligence curve actually needs
- Fixed-quality comparisonCost measured at constant task quality, context, latency, and availability.
- Outcome-based accountingCost per accepted result, not cost per generated token.
- Right-sized routingEasy work handled by the cheapest model that reliably meets the bar.
- Efficient servingBatching and scheduling that convert cheap compute into cheap service.
- Scalable verificationEvaluation and tool execution that keep pace with falling token prices.
The price curve, the energy floor, and the task bill
Epoch AI's September 2026 analysis estimates that the price of a fixed level of performance across five benchmarks fell about 47% per quarter since 2023, or roughly 13-fold per year. This is an observed API price trend across selected tasks and thresholds, not a claim that provider cost, every capability, or an entire useful task falls at the same rate. Epoch's benchmark-level data expose the task and threshold differences hidden by one headline average.
Google measured 0.24 Wh for the median Gemini Apps text prompt in May 2025 on its stated serving boundary. At an illustrative $0.10/kWh, the associated electricity is $0.000024 per median prompt, or $24 per million such prompts. That is an electricity component for one product and workload, not a universal thermodynamic minimum or an API price. Chips, memory, cooling, networking, training recovery, and human review add to the bill; longer reasoning calls can use more energy.
For an accepted outcome, the relevant identity is cost per attempt × attempts per accepted result + verification cost. A system succeeding on half its independent attempts needs two attempts on average, doubling the inference component before review or tools. Benchmark success rates and pass@k should be reported with their task definition and retry policy. Falling cost per token can therefore coexist with rising spending when workloads grow or use more reasoning. Epoch's inference-economics analysis estimates major-provider inference revenue growing at least threefold annually, a demand signal distinct from unit price.
The supply-side bill can move in the opposite direction. Epoch estimated frontier final-run hardware and energy costs growing 2.4-fold annually from 2016. Final-run cost is neither total research spend nor the amortized cost of one API request; both utilization and lifetime demand determine how it is recovered.
Sources, method, and boundaries
Price and benchmark results are measured but use different boundaries. No API list price is treated as underlying production cost. The interactive calculator uses representative September 2026 published API pricing across budget, mainstream, and flagship tiers as an illustration of the article's own compute-plus-verification frame, not a real-time price quote for any named provider.
- MMLU
- A multi-task benchmark measuring language-model accuracy across academic and professional subjects.
- Accepted outcome
- A model response that passes whatever verification or human review the application requires before use.
- Quantization
- Reducing the numerical precision used to store and compute model weights, lowering memory and compute cost.
