Will Intelligence Become Too Cheap to Meter?

The price of a given level of AI has fallen more than a hundredfold in two years. But people keep asking for smarter answers, and checking that an answer is right can cost more than producing it.

Last updated September 2026
Figure 1 · The collapse

Yesterday's genius is today's bargain

In 1954, the head of the U.S. Atomic Energy Commission promised electricity “too cheap to meter.” It never happened. AI is making a similar promise, and on one measure it is already coming true faster than anyone expected: whatever the best model could do two years ago now costs pennies.

>280×Fall in the price of GPT-3.5-level performance from late 2022 to late 2024, per Stanford's AI Index.
142×How much smaller the smallest model scoring over 60% on a standard knowledge test became over the same period.
3.3×Gain in the best inference speed on a standard 70-billion-parameter model in one year of MLPerf results.

API price is not provider cost; MMLU is not every useful task; benchmark throughput is not interactive service quality.

The story in one paragraph

Two curves are running at once. On the first, any fixed level of AI ability gets dramatically cheaper every year, as smaller models, better chips and clever serving shrink what it costs to reach it. On the second, the frontier keeps moving: new models think longer, read more, use tools and cost more to train. People do not want yesterday's answers; they want today's, so total spending on AI is rising even as each unit falls in price. And there is a third cost the price charts leave out: an answer is only worth something if it is right. Checking, retrying and human review increasingly dominate the bill. So the honest answer is yes for routine intelligence, no for the frontier, and the real unit to watch is the cost of an answer you can trust.

  • Fixed ability is collapsing in price: Epoch AI estimates about 13-fold per year since 2023.
  • Models are shrinking: the smallest model passing a 60% knowledge bar went from 540 billion parameters to 3.8 billion in two years.
  • The frontier is getting pricier: the cost of a frontier training run has grown about 2.4-fold per year.
  • Trust is the new cost: if a system gets it right half the time, every accepted answer costs at least twice as much.

Measured results, derived quantities, projections, targets, and editorial inference are identified by context. Announced capacity is never treated as operating performance.

Part I: What you are actually paying for

Tokens are cheap. Accepted answers are the real unit.

A price chart only means something if the quality stays fixed. Cheap tokens that are wrong, slow or unusable are not progress. Use the calculator to see how retries and review change the true cost of a finished task.

Figure 2 · Interactive input model

What does an accepted answer actually cost?

The model fixes a task, then separates raw token cost from the verification and retry overhead needed to trust the answer, echoing the article's own frame: compute plus memory plus serving, divided by accepted task outcomes.

Budget model, low-stakes task$0.0002/accepted outcome

$0.0002 in raw tokens × 1.1× verification overhead

Raw token cost
$0.0002
Verification & retry overhead
1.1×
Cost per 1,000 accepted outcomes
$0.19

September 2026 API pricing spans roughly $0.02–0.20 per million input tokens for budget and mainstream models up to $5–10 for flagship models, with output typically priced 5–6× higher than input. Move the verification slider and notice how a 25× sticker-price gap between budget and flagship models can widen to far more once retries and review are counted, the mechanism behind the article's claim that cheap tokens push scarcity into trustworthy evaluation.

Calculation and boundaries

Raw token cost = (prompt tokens ÷ 1,000,000 × input price) + (output tokens ÷ 1,000,000 × output price). Cost per accepted outcome = raw token cost × verification/retry overhead multiplier. Excludes serving infrastructure cost, batching discounts, caching, and the underlying provider cost of compute: API list price is not production cost. Verification overhead is an illustrative multiplier representing retries, tool calls, and human review, not a benchmarked figure for any specific task.

An editorial illustration of the article's own cost-of-intelligence frame, not a price quote for any named model or provider.

The price curve, the energy bill, and the task bill

Epoch AI's September 2026 analysis estimates that the price of a fixed level of performance across five benchmarks fell about 47% per quarter since 2023, roughly 13-fold per year. That is an observed API price trend on selected tasks, not proof that provider costs or every capability fall at the same rate; Epoch's benchmark-level data show how much the rate varies by task.

The electricity behind a single answer is already tiny. Google measured 0.24 Wh for the median Gemini Apps text prompt in May 2025, about the energy of running a microwave for one second. At $0.10/kWh, that is $0.000024 per prompt, or $24 per million. Chips, memory, cooling, networking and training costs add far more, and long reasoning calls use more energy than a median prompt.

The bill that matters is cost per attempt × attempts per accepted result + verification cost. A system that succeeds half the time needs two attempts on average, doubling the inference cost before anyone checks the work. This is why falling prices per token can coexist with rising spending: Epoch estimates major-provider inference revenue growing at least threefold annually.

Part II: Where the cost hides

Four layers between a model and a trusted answer

Cheap arithmetic does not automatically become a cheap answer. Each layer above the model adds its own cost, and the top one, deciding whether to trust the result, is often the most expensive.

01

Model and data

The model's design, its training data, fine-tuning and any documents it can look up decide how good an answer can be.

Measure
Task quality · robustness
Failure boundary
Gaps in data and testing cap quality no matter how cheaply the model runs.
Where the frontier moves

Training and retrieval methods that raise quality without making the model bigger.

02

Numbers and memory

How precisely weights are stored and how fast they move from memory decide how much each word costs to generate.

Measure
Bytes/token · joules/token
Failure boundary
Memory bandwidth, not raw compute, usually limits how fast a model can talk.
Where the frontier moves

Lower-precision numbers and sparse models that move fewer bytes per word.

03

Serving system

Batching many users together, routing requests and predicting several words at once trade speed against cost.

Measure
Tokens/s · tail latency
Failure boundary
Cheap chips only mean cheap answers if they stay busy.
Where the frontier moves

Serving tricks that keep chips busy without making users wait.

04

Application

Tools, checks, human review and workflow design decide whether an answer actually gets used.

Measure
Cost per accepted outcome
Failure boundary
A cheap wrong answer can create a very expensive mistake.
Where the frontier moves

Verification that catches errors without redoing the whole task.

Part III: The frontier keeps moving

Cheaper answers create hungrier questions

Every time AI gets cheaper, people ask it to do more: read whole codebases, reason for minutes, act as an agent across many tools. Meanwhile, Epoch estimates frontier training-run costs growing 2.4-fold a year since 2016. Like cheaper lighting in the nineteenth century, cheaper intelligence has so far meant using much more of it, not spending less.

compute + memory + serving÷accepted task outcomes=cost of intelligence
Part IV: How the cost keeps falling

The bottleneck moves from generating to verifying

As raw tokens get cheap, the scarce resources become trustworthy evaluation, reliable tool use and human attention. The industry is attacking both sides at once.

Smaller specialists

Send each task to the cheapest model that reliably gets it right, not the biggest one available.

Better serving

Batch, cache and schedule requests so chips stay busy while users still get fast replies.

Hardware-software codesign

Match number formats and memory layout to the actual workload.

Outcome evaluation

Measure work that was accepted and the cost of errors, not words generated or benchmark scores.

Who is building what

Inference chips, speculative decoding, new attention designs, 4- and 8-bit number formats and specialized compilers all push down the cost per token. Search the record, or filter by efficiency layer.

8 programmes
NVIDIATensorRT-LLM & Blackwell NVL72Second-generation Transformer Engines supporting native microscopic FP4 and FP8 quantization, paired with TensorRT-LLM in-flight batching and KV-cache compression
Reported evidence
Dominant market platform powering hyperscale AI inference; demonstrated up to 30× throughput speedup on Blackwell versus earlier architectures.
Announced next step
Driving inference cost per token down by orders of magnitude while supporting 100k+ token real-time context generation.
Unresolved risk
High accelerator capital acquisition costs, 120 kW rack thermal power envelopes, and software ecosystem vendor lock-in via CUDA.
GroqLanguage Processing Unit (LPU)Deterministic, single-core Tensor Streaming Processor (TSP) architecture with massive on-chip SRAM (230 MB per chip) and zero off-chip DRAM latency
Reported evidence
Commercial cloud API delivering industry-record generation speeds exceeding 500–800 tokens per second on open models (Llama-3).
Announced next step
Ultra-low-latency real-time voice, code generation, and multi-turn agentic loops requiring immediate token streaming.
Unresolved risk
Extremely high silicon area per gigabyte of memory (SRAM density limits) requiring hundreds of interconnected chips to hold a single large model.
Cerebras SystemsCS-3 Wafer-Scale EngineWafer-scale single-piece silicon processor with 900,000 AI cores and 44 GB of on-wafer SRAM delivering 21 petabytes/second memory bandwidth
Reported evidence
Commercial inference service setting world records for token generation velocity on open frontier models, outperforming multi-GPU clusters.
Announced next step
Replacing sprawling GPU clusters with single-wafer compute nodes for extreme-speed inference serving.
Unresolved risk
Custom packaging, power delivery, and water-cooling infrastructure required for a single 23 kW wafer-scale processor package.
DeepSeekDeepSeek-V3 & R1 (MLA)Multi-head Latent Attention (MLA) compressing the KV-cache by an order of magnitude, paired with fine-grained Mixture-of-Experts (MoE) activating only ~37B out of 671B parameters
Reported evidence
Trained frontier-grade reasoning models for a fraction of traditional compute costs; released models delivering commercial API pricing well below Western frontier APIs.
Announced next step
Democratizing high-capability reasoning and coding models through algorithmic and architectural efficiency rather than brute-force scaling.
Unresolved risk
Quantization degradation on complex algorithmic reasoning tasks, and long-term hardware optimization on non-NVIDIA silicon.
Together AI / vLLM ProjectPagedAttention & FlashAttentionVirtual memory management for KV-caches (PagedAttention) eliminating memory fragmentation, combined with speculative decoding and kernel fusion
Reported evidence
vLLM is the ubiquitous open-source standard powering commercial cloud inference, increasing serving concurrency and throughput by 2–4× on identical hardware.
Announced next step
Universal low-latency inference serving software running seamlessly across heterogeneous silicon architectures.
Unresolved risk
Managing memory fragmentation and scheduling overhead under bursty, highly variable prompt length distributions.
Mistral AISmall & MoE ModelsHigh-efficiency open-weights models and dense/MoE architectures (Mixtral 8x7B/8x22B) optimized for high parameter efficiency and distillation
Reported evidence
Models match or exceed earlier generation flagship model capabilities on common benchmarks at a small fraction of active parameter count and inference cost.
Announced next step
Edge-deployable and on-premise high-throughput models running efficiently on commodity enterprise servers.
Unresolved risk
Performance ceilings on highly complex multi-step mathematical and software architecture planning versus trillion-parameter dense models.
ModularMAX Engine & Mojo CompilerNext-generation unified AI compiler and runtime (MAX) and systems language (Mojo) optimizing kernel execution and memory management across heterogeneous CPUs and GPUs
Reported evidence
Commercial benchmarks demonstrate significant throughput speedups on PyTorch and TensorFlow models without manual CUDA C++ kernel rewrites.
Announced next step
Decoupling AI application software completely from underlying semiconductor hardware architectures.
Unresolved risk
Competing against heavily entrenched proprietary optimization libraries (NVIDIA TensorRT, AMD ROCm) backed by massive ecosystem inertia.
MLCommonsMLPerf Inference BenchmarksOpen industry consortium standardizing rigorous benchmarking of AI model inference across latency, throughput, energy efficiency, and cost
Reported evidence
Bi-annual published results independently measuring performance across server, edge, and mobile accelerators from all major semiconductor vendors.
Announced next step
Providing unvarnished, reproducible price-performance transparency across cloud and on-premise AI acceleration hardware.
Unresolved risk
Benchmark gaming through non-representative optimizations (extreme pruning, unrealistic batch sizes) that do not reflect production deployment.

Advertised tokens-per-dollar metrics often assume batch processing on small quantized models; interactive low-latency serving with long context windows incurs heavy KV-cache memory overhead.

The optimistic view, with conditions

Intelligence becomes a utility with two tiers

Routine intelligence, the kind that drafts, summarizes, translates and answers, is heading toward near-zero cost and will be built into everything. The premium will shift to hard, high-stakes work where people pay for evidence that the answer is right.

Commodity tier

Yesterday's frontier keeps falling

Distillation and better serving push old capability onto smaller, cheaper systems every year.

Premium tier

Spend concentrates on trust

The hardest tasks pay for reliability and verification, not raw tokens.

System level

Measure the accepted outcome

Cost per accepted result, not cost per token, tells us whether the curve is real progress.

What cheap, useful intelligence actually needs

  1. Fair comparisonsCost measured at fixed quality, context, speed and availability.
  2. Outcome accountingCost per accepted result, not per word generated.
  3. Right-sized routingEasy work handled by the cheapest model that reliably gets it right.
  4. Efficient servingBatching and scheduling that turn cheap compute into cheap service.
  5. Scalable verificationChecking that keeps pace with falling token prices.

Sources, method, and boundaries

Price and benchmark results are measured but use different boundaries. No API list price is treated as underlying production cost. Final-run training cost is neither total research spend nor the amortized cost of one request. The interactive calculator uses representative September 2026 published API pricing across budget, mainstream, and flagship tiers as an illustration of the article's own compute-plus-verification frame, not a real-time price quote for any named provider.

MMLU
A multi-task benchmark measuring language-model accuracy across academic and professional subjects.
Accepted outcome
A model response that passes whatever verification or human review the application requires before use.
Quantization
Reducing the numerical precision used to store and compute model weights, lowering memory and compute cost.

Read More

Essays, books, talks, and research that shaped this field’s arguments. Influence is not endorsement; company communications and advocacy are labeled. Some publisher links require a subscription.

  1. Lab-CEO essay

    Dario Amodei — Machines of Loving Grace

    The capability vision from a lab CEO, to pair with measured task quality and the cost of checking work.

  2. Founder essay

    Sam Altman — Moore’s Law for Everything

    The earlier abundance argument linking AI progress, falling costs, and the distribution of gains.

  3. Industry measurement collection

    SemiAnalysis — Models and Research

    Inference economics and hardware analysis to compare against token prices and accepted-task costs.

  4. Measurement essay

    Epoch AI, “The plunging price of thought” (2026)

    A data-driven essay/report measuring the rapid fall in cost for a fixed level of AI performance rather than simply tracking token prices.

  5. Essay

    Rich Sutton, “The Bitter Lesson” (2019)

    One of the most influential essays in modern AI, arguing that general methods leveraging increasing computation tend to dominate hand-built domain knowledge over time.

Across the fields: learning curves, deployment, and rebound