Nvidia's $20 billion licensing deal with Groq marks a watershed moment in artificial intelligence: the company is betting that specialized inference chips will define the next era of AI profitability, not raw training power. In December 2025, Nvidia formalized this vision by acquiring Groq's technology and talent, and now Samsung will manufacture the Groq 3 Language Processing Unit (LPU) starting in the third quarter of 2026, with shipments expected to begin shortly after .

The shift reflects a fundamental truth about modern AI economics. When OpenAI trained GPT-4 in early 2023, it required roughly 25,000 Nvidia A100 graphics processing units (GPUs) running continuously for over three months. But training is a one-time event. Inference, the process of running a trained model to generate responses, happens continuously and at massive scale. Microsoft Azure, which powered ChatGPT from the start, quickly discovered it needed far more computing power to serve users than it had needed to train the model in the first place .

Why Groq's Approach to Inference Is Fundamentally Different?

Groq's LPU design sidesteps the traditional memory bottleneck that has constrained inference performance. Instead of relying on High Bandwidth Memory (HBM), which is expensive and power-hungry, Groq's architecture uses approximately 500 megabytes of on-die Static Random Access Memory (SRAM) with a bandwidth of roughly 150 terabytes per second. That is seven times the memory bandwidth that Nvidia's Rubin platform offers, enabling dramatically faster token generation . Tokens are the small units of text or data that AI models process; the speed at which a system generates tokens directly determines how quickly users see responses.

The Groq 3 LPU will function as a decode-phase co-processor within Nvidia's Vera Rubin AI platform, integrated into 256-chip inference racks designed for ultra-low latency performance . The decode phase is when the model generates output tokens one at a time, and this is where inference efficiency matters most. By pairing Groq's SRAM-based design with Samsung's advanced 4-nanometer manufacturing process, Nvidia aims to deliver inference solutions that are not just faster but dramatically more cost-effective at scale .

How to Understand the Economics Driving This Shift?

• Token Economics: The industry now measures performance using "tokenomics," which means tokens generated per second per watt of power consumed. This metric directly translates to cost per token delivered to end users, making it the primary competitive benchmark for inference hardware.
• Continuous Demand: Inference demand has grown approximately one millionfold over the past two years, driven by agentic AI frameworks (autonomous AI agents that execute complex workflows independently), reasoning models, and the sheer number of daily users accessing AI services .
• Enterprise-Led Growth: The majority of this inference demand comes from enterprise customers, a trend that is expected to accelerate as companies deploy AI agents for internal operations and customer-facing services .

The scale of Nvidia's ambition is staggering. The company projects $1 trillion in platform orders through 2027, double its earlier forecast, with the Groq 3 LPU positioned as a cornerstone of that growth . For context, this reflects confidence that inference workloads will dominate AI infrastructure spending in the coming years, not training.

What Does This Mean for the Broader AI Hardware Landscape?

Microsoft's own inference accelerator, the Azure Maia 200, illustrates how the market is fragmenting. Rather than a single "best" chip for all AI tasks, cloud providers are adopting heterogeneous infrastructure, meaning they deploy different specialized chips for different workloads. Maia 200 sits between a general-purpose GPU and a highly specialized chip like Groq's LPU, offering a middle ground that preserves flexibility while optimizing for known inference patterns .

"Maia 200 sits somewhere in the middle of a generalized parallel processor like a GPU and a specialized chip such as Cerebras' CS-3 and Groq's Language-Processing Unit," explained Andrew Wall, General Manager of Azure Maia at Microsoft.

Andrew Wall, General Manager of Azure Maia, Microsoft

This middle-ground strategy reflects a real constraint: model architectures, context lengths (the amount of text a model can process at once), and serving patterns are still evolving too rapidly for hard specialization to be a safe universal bet. However, Nvidia's acquisition of Groq suggests the company believes that inference-focused architectures have matured enough to justify deep specialization .

The manufacturing partnership with Samsung is equally significant. Samsung will produce the Groq 3 LPU on its 4-nanometer process, the same advanced node used for cutting-edge mobile and data center chips. This signals that inference hardware has become a first-class priority in semiconductor manufacturing, competing for wafer capacity alongside consumer electronics and other high-margin products .

Demand for inference capacity is so intense that it is straining the entire semiconductor supply chain. A single gigawatt of inference capacity using Nvidia's Vera Rubin platform requires approximately 55,000 wafer starts per month on the 3-nanometer process, 6,000 on the 5-nanometer process, and 170,000 DRAM wafer starts monthly . The bottleneck is not chip design but manufacturing capacity, particularly extreme ultraviolet (EUV) lithography equipment. ASML, the only company that makes EUV machines, sold roughly 48 units in 2025 at prices between $200 million and $400 million each, and production capacity is difficult to scale .

For enterprises and cloud providers, the Groq 3 LPU represents a concrete path to lower inference costs. The chip's architecture is optimized for the specific computational patterns that dominate inference workloads, meaning it can deliver more tokens per watt than general-purpose hardware. As inference demand continues to grow, this efficiency advantage translates directly to lower operational costs and faster response times for end users .

The Samsung partnership also provides a hedge against supply constraints. By diversifying manufacturing beyond Taiwan Semiconductor Manufacturing Company (TSMC), Nvidia reduces its exposure to geopolitical risk and capacity bottlenecks at a single foundry. Samsung's commitment to producing the Groq 3 LPU signals confidence that inference hardware will remain a high-priority, high-margin product for years to come .

What happens next will depend on how quickly Samsung can ramp production and how aggressively enterprises adopt Nvidia's Vera Rubin inference racks. The company projects shipments to begin in Q3 2026, which would put the Groq 3 LPU in customer hands by mid-year. If adoption matches Nvidia's expectations, the Groq 3 LPU could redefine inference efficiency at scale and further cement Nvidia's dominance in the trillion-dollar AI hardware market .