Tokens are cheap. Accepted answers are the real unit.
A price chart only means something if the quality stays fixed. Cheap tokens that are wrong, slow or unusable are not progress. Use the calculator to see how retries and review change the true cost of a finished task.
What does an accepted answer actually cost?
The model fixes a task, then separates raw token cost from the verification and retry overhead needed to trust the answer, echoing the article's own frame: compute plus memory plus serving, divided by accepted task outcomes.
$0.0002 in raw tokens × 1.1× verification overhead
- Raw token cost
- $0.0002
- Verification & retry overhead
- 1.1×
- Cost per 1,000 accepted outcomes
- $0.19
September 2026 API pricing spans roughly $0.02–0.20 per million input tokens for budget and mainstream models up to $5–10 for flagship models, with output typically priced 5–6× higher than input. Move the verification slider and notice how a 25× sticker-price gap between budget and flagship models can widen to far more once retries and review are counted, the mechanism behind the article's claim that cheap tokens push scarcity into trustworthy evaluation.
Calculation and boundaries
Raw token cost = (prompt tokens ÷ 1,000,000 × input price) + (output tokens ÷ 1,000,000 × output price). Cost per accepted outcome = raw token cost × verification/retry overhead multiplier. Excludes serving infrastructure cost, batching discounts, caching, and the underlying provider cost of compute: API list price is not production cost. Verification overhead is an illustrative multiplier representing retries, tool calls, and human review, not a benchmarked figure for any specific task.
The price curve, the energy bill, and the task bill
Epoch AI's September 2026 analysis estimates that the price of a fixed level of performance across five benchmarks fell about 47% per quarter since 2023, roughly 13-fold per year. That is an observed API price trend on selected tasks, not proof that provider costs or every capability fall at the same rate; Epoch's benchmark-level data show how much the rate varies by task.
The electricity behind a single answer is already tiny. Google measured 0.24 Wh for the median Gemini Apps text prompt in May 2025, about the energy of running a microwave for one second. At $0.10/kWh, that is $0.000024 per prompt, or $24 per million. Chips, memory, cooling, networking and training costs add far more, and long reasoning calls use more energy than a median prompt.
The bill that matters is cost per attempt × attempts per accepted result + verification cost. A system that succeeds half the time needs two attempts on average, doubling the inference cost before anyone checks the work. This is why falling prices per token can coexist with rising spending: Epoch estimates major-provider inference revenue growing at least threefold annually.
Four layers between a model and a trusted answer
Cheap arithmetic does not automatically become a cheap answer. Each layer above the model adds its own cost, and the top one, deciding whether to trust the result, is often the most expensive.
Model and data
The model's design, its training data, fine-tuning and any documents it can look up decide how good an answer can be.
- Measure
- Task quality · robustness
- Failure boundary
- Gaps in data and testing cap quality no matter how cheaply the model runs.
Where the frontier moves
Training and retrieval methods that raise quality without making the model bigger.
Numbers and memory
How precisely weights are stored and how fast they move from memory decide how much each word costs to generate.
- Measure
- Bytes/token · joules/token
- Failure boundary
- Memory bandwidth, not raw compute, usually limits how fast a model can talk.
Where the frontier moves
Lower-precision numbers and sparse models that move fewer bytes per word.
Serving system
Batching many users together, routing requests and predicting several words at once trade speed against cost.
- Measure
- Tokens/s · tail latency
- Failure boundary
- Cheap chips only mean cheap answers if they stay busy.
Where the frontier moves
Serving tricks that keep chips busy without making users wait.
Application
Tools, checks, human review and workflow design decide whether an answer actually gets used.
- Measure
- Cost per accepted outcome
- Failure boundary
- A cheap wrong answer can create a very expensive mistake.
Where the frontier moves
Verification that catches errors without redoing the whole task.
Cheaper answers create hungrier questions
Every time AI gets cheaper, people ask it to do more: read whole codebases, reason for minutes, act as an agent across many tools. Meanwhile, Epoch estimates frontier training-run costs growing 2.4-fold a year since 2016. Like cheaper lighting in the nineteenth century, cheaper intelligence has so far meant using much more of it, not spending less.
The bottleneck moves from generating to verifying
As raw tokens get cheap, the scarce resources become trustworthy evaluation, reliable tool use and human attention. The industry is attacking both sides at once.
Smaller specialists
Send each task to the cheapest model that reliably gets it right, not the biggest one available.
Better serving
Batch, cache and schedule requests so chips stay busy while users still get fast replies.
Hardware-software codesign
Match number formats and memory layout to the actual workload.
Outcome evaluation
Measure work that was accepted and the cost of errors, not words generated or benchmark scores.
Who is building what
Inference chips, speculative decoding, new attention designs, 4- and 8-bit number formats and specialized compilers all push down the cost per token. Search the record, or filter by efficiency layer.
NVIDIATensorRT-LLM & Blackwell NVL72Second-generation Transformer Engines supporting native microscopic FP4 and FP8 quantization, paired with TensorRT-LLM in-flight batching and KV-cache compression
- Reported evidence
- Dominant market platform powering hyperscale AI inference; demonstrated up to 30× throughput speedup on Blackwell versus earlier architectures.
- Announced next step
- Driving inference cost per token down by orders of magnitude while supporting 100k+ token real-time context generation.
- Unresolved risk
- High accelerator capital acquisition costs, 120 kW rack thermal power envelopes, and software ecosystem vendor lock-in via CUDA.
GroqLanguage Processing Unit (LPU)Deterministic, single-core Tensor Streaming Processor (TSP) architecture with massive on-chip SRAM (230 MB per chip) and zero off-chip DRAM latency
- Reported evidence
- Commercial cloud API delivering industry-record generation speeds exceeding 500–800 tokens per second on open models (Llama-3).
- Announced next step
- Ultra-low-latency real-time voice, code generation, and multi-turn agentic loops requiring immediate token streaming.
- Unresolved risk
- Extremely high silicon area per gigabyte of memory (SRAM density limits) requiring hundreds of interconnected chips to hold a single large model.
Cerebras SystemsCS-3 Wafer-Scale EngineWafer-scale single-piece silicon processor with 900,000 AI cores and 44 GB of on-wafer SRAM delivering 21 petabytes/second memory bandwidth
- Reported evidence
- Commercial inference service setting world records for token generation velocity on open frontier models, outperforming multi-GPU clusters.
- Announced next step
- Replacing sprawling GPU clusters with single-wafer compute nodes for extreme-speed inference serving.
- Unresolved risk
- Custom packaging, power delivery, and water-cooling infrastructure required for a single 23 kW wafer-scale processor package.
DeepSeekDeepSeek-V3 & R1 (MLA)Multi-head Latent Attention (MLA) compressing the KV-cache by an order of magnitude, paired with fine-grained Mixture-of-Experts (MoE) activating only ~37B out of 671B parameters
- Reported evidence
- Trained frontier-grade reasoning models for a fraction of traditional compute costs; released models delivering commercial API pricing well below Western frontier APIs.
- Announced next step
- Democratizing high-capability reasoning and coding models through algorithmic and architectural efficiency rather than brute-force scaling.
- Unresolved risk
- Quantization degradation on complex algorithmic reasoning tasks, and long-term hardware optimization on non-NVIDIA silicon.
Together AI / vLLM ProjectPagedAttention & FlashAttentionVirtual memory management for KV-caches (PagedAttention) eliminating memory fragmentation, combined with speculative decoding and kernel fusion
- Reported evidence
- vLLM is the ubiquitous open-source standard powering commercial cloud inference, increasing serving concurrency and throughput by 2–4× on identical hardware.
- Announced next step
- Universal low-latency inference serving software running seamlessly across heterogeneous silicon architectures.
- Unresolved risk
- Managing memory fragmentation and scheduling overhead under bursty, highly variable prompt length distributions.
Mistral AISmall & MoE ModelsHigh-efficiency open-weights models and dense/MoE architectures (Mixtral 8x7B/8x22B) optimized for high parameter efficiency and distillation
- Reported evidence
- Models match or exceed earlier generation flagship model capabilities on common benchmarks at a small fraction of active parameter count and inference cost.
- Announced next step
- Edge-deployable and on-premise high-throughput models running efficiently on commodity enterprise servers.
- Unresolved risk
- Performance ceilings on highly complex multi-step mathematical and software architecture planning versus trillion-parameter dense models.
ModularMAX Engine & Mojo CompilerNext-generation unified AI compiler and runtime (MAX) and systems language (Mojo) optimizing kernel execution and memory management across heterogeneous CPUs and GPUs
- Reported evidence
- Commercial benchmarks demonstrate significant throughput speedups on PyTorch and TensorFlow models without manual CUDA C++ kernel rewrites.
- Announced next step
- Decoupling AI application software completely from underlying semiconductor hardware architectures.
- Unresolved risk
- Competing against heavily entrenched proprietary optimization libraries (NVIDIA TensorRT, AMD ROCm) backed by massive ecosystem inertia.
MLCommonsMLPerf Inference BenchmarksOpen industry consortium standardizing rigorous benchmarking of AI model inference across latency, throughput, energy efficiency, and cost
- Reported evidence
- Bi-annual published results independently measuring performance across server, edge, and mobile accelerators from all major semiconductor vendors.
- Announced next step
- Providing unvarnished, reproducible price-performance transparency across cloud and on-premise AI acceleration hardware.
- Unresolved risk
- Benchmark gaming through non-representative optimizations (extreme pruning, unrealistic batch sizes) that do not reflect production deployment.
Advertised tokens-per-dollar metrics often assume batch processing on small quantized models; interactive low-latency serving with long context windows incurs heavy KV-cache memory overhead.
The optimistic view, with conditions
Intelligence becomes a utility with two tiers
Routine intelligence, the kind that drafts, summarizes, translates and answers, is heading toward near-zero cost and will be built into everything. The premium will shift to hard, high-stakes work where people pay for evidence that the answer is right.
Yesterday's frontier keeps falling
Distillation and better serving push old capability onto smaller, cheaper systems every year.
Spend concentrates on trust
The hardest tasks pay for reliability and verification, not raw tokens.
Measure the accepted outcome
Cost per accepted result, not cost per token, tells us whether the curve is real progress.
What cheap, useful intelligence actually needs
- Fair comparisonsCost measured at fixed quality, context, speed and availability.
- Outcome accountingCost per accepted result, not per word generated.
- Right-sized routingEasy work handled by the cheapest model that reliably gets it right.
- Efficient servingBatching and scheduling that turn cheap compute into cheap service.
- Scalable verificationChecking that keeps pace with falling token prices.
Sources, method, and boundaries
Price and benchmark results are measured but use different boundaries. No API list price is treated as underlying production cost. Final-run training cost is neither total research spend nor the amortized cost of one request. The interactive calculator uses representative September 2026 published API pricing across budget, mainstream, and flagship tiers as an illustration of the article's own compute-plus-verification frame, not a real-time price quote for any named provider.
- MMLU
- A multi-task benchmark measuring language-model accuracy across academic and professional subjects.
- Accepted outcome
- A model response that passes whatever verification or human review the application requires before use.
- Quantization
- Reducing the numerical precision used to store and compute model weights, lowering memory and compute cost.
Read More
Essays, books, talks, and research that shaped this field’s arguments. Influence is not endorsement; company communications and advocacy are labeled. Some publisher links require a subscription.
- Lab-CEO essay
Dario Amodei — Machines of Loving Grace
The capability vision from a lab CEO, to pair with measured task quality and the cost of checking work.
- Founder essay
Sam Altman — Moore’s Law for Everything
The earlier abundance argument linking AI progress, falling costs, and the distribution of gains.
- Industry measurement collection
SemiAnalysis — Models and Research
Inference economics and hardware analysis to compare against token prices and accepted-task costs.
- Measurement essay
Epoch AI, “The plunging price of thought” (2026)
A data-driven essay/report measuring the rapid fall in cost for a fixed level of AI performance rather than simply tracking token prices.
- Essay
Rich Sutton, “The Bitter Lesson” (2019)
One of the most influential essays in modern AI, arguing that general methods leveraging increasing computation tend to dominate hand-built domain knowledge over time.
Across the fields: learning curves, deployment, and rebound
- Theodore Wright — Factors Affecting the Cost of Airplanes (1936)
The original experience-curve formulation: costs change with accumulated production.
- Kenneth Arrow — The Economic Implications of Learning by Doing (1962)
The economics of productivity improvement through production experience.
- William Stanley Jevons — The Coal Question (1865)
The classic rebound argument: lower effective costs can expand total demand.
- Arnulf Grübler — The costs of the French nuclear scale-up: A case of negative learning by doing (2010)
A counterexample to assuming that greater deployment always lowers costs.



















