The custom silicon category is growing at a compound annual rate of 44.6 percent, compared to 16.1 percent for general-purpose GPUs. That gap will close some as custom silicon matures, but the direction is clear: the inference stack is being unbundled, and specialized hardware is winning efficiency arguments that general-purpose chips can't answer.
Speed That Rewrites Economics
Nowhere is this more visible than in the performance numbers coming out of inference-first hardware.
Groq, for instance, delivers 300 tokens per second on Llama 70B, roughly 10 times faster than an H100 on the same workload. At that throughput, the economics of interactive AI applications become suddenly tractable. A chatbot that would cost /bin/bash.003 per message on H100 becomes a fraction of that on inference-optimized silicon. When you're processing millions of requests per day, that arithmetic changes product strategies.
The cost trajectory for inference has been nothing short of astonishing. Across the industry, inference costs have dropped 280-fold in just two years. That decline is partly hardware, partly software optimization, and partly the predictable result of scale. But it's also a direct consequence of the efficiency pivot: companies are no longer content to run inference on training-grade hardware, and the hardware ecosystem is responding.
There's a pricing innovation reinforcing the trend. Flat-rate models, which offer unlimited or high-volume access for a fixed monthly fee, are now roughly three times cheaper than per-token pricing at moderate to high usage levels. For any organization running inference at scale, the flat-rate structure eliminates per-query anxiety and rewards the kind of steady, heavy usage that characterizes production AI. This is a signal that the market expects inference costs to keep falling and wants customers locked into volume, not marginal cost, calculations.
Who's Spending and Why It Matters
The scale of investment behind this shift is staggering. The hyperscalers, Microsoft, Amazon, Google, Meta, are on track to spend a combined 00 billion on capital expenditures in 2026, with a substantial portion directed at AI infrastructure. That's not a bet on today's models. That's a bet on a world where inference is the dominant compute workload and efficiency is the primary competitive variable.
Indeed, inference has already crossed a threshold. For most large-scale AI deployments, inference now accounts for 60 to 70 percent of total compute spend. Training costs get the headlines, but the day-to-day expense of running models in the wild is where the money actually goes. This is why the efficiency pivot matters so much: shaving 30 percent off inference costs compounds across every request, every user, every product that touches a language model.
The allocation decisions being made in hyperscaler boardrooms right now will determine which silicon vendors thrive and which fade. The era of treating all AI workloads the same is ending. Training and inference are diverging into distinct engineering disciplines with distinct hardware preferences, and the companies that recognize that earliest will be best positioned to capture value.
What Remains Scarce
So what is the real scarcity now that GPUs are no longer the chokepoint?
Talent, obviously, the ability to build, optimize, and operate inference at scale is rare and getting rarer.
But more specifically, the scarce resource is the knowledge of where to spend compute budget for maximum impact. Raw hardware is available. The question is how to deploy it: which models, on which silicon, for which use cases, at what batch size and precision, with what serving infrastructure. That optimization stack is where the competitive moats are being dug now.
The GPU shortage was loud and visible. The new scarcity is quieter, more technical, and harder to shortcut. That's probably why it replaced the old one.
The chips are back. The race just changed tracks.
